On 10/21/2018 2:08 PM, Walter Bright wrote:
> On 10/21/2018 12:20 PM, Nicholas Wilson wrote:
>> Yes, but the problem you describe is arises from implicit conversion in the other direction, which is not part of the proposal.
> 
> It's Manu's example.

Then I don't know what the proposal is. Pieces of it appear to be scattered over numerous posts, mixed in with other text, opinions, and handwavy stuff. There's nothing to point to that is "the proposal".

I suggest you and Manu write up a proper proposal. Something that is complete, has nothing else in it, has a rationale, illuminating examples, and explains why alternatives are inferior.

For examples of how to do it:

https://github.com/dlang/DIPs/tree/master/DIPs

Trying to rewrite the semantics of shared is not a simple task, doing multithreading correctly is a minefield of "OOPS! I didn't think of that!" and if anything cries out for a DIP, your and Manu's proposal does.

On Sun, 21 Oct 2018 12:04:16 -0700, Manu wrote:
> On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>> Note that there may well be a good way to get the good properties of MP without breaking the type system, but MP itself is not good because it breaks @safe.
> 
> Show me. Nobody has been able to show that yet. I'd really like to know this.

If we only used your proposal and only used @safe code, we wouldn't have any data races, but that's only because we wouldn't have any shared data. We'd have shared *variables*, but they would not contain any data we could read or alter, and that's pretty much useless.

To use your proposal, we need to cast data back from shared to unshared. When it's unshared, we need to make sure that exactly one thread has a reference to that data as unshared. And @safe *should* help us with that. Currently, it helps because casting unshared to shared is not @safe, because it makes it trivial to get multiple threads with unshared references to the same data. And that's when you're using shared as expected rather than doing something weird.

On Sunday, 21 October 2018 at 21:32:14 UTC, Walter Bright wrote:
> On 10/21/2018 2:08 PM, Walter Bright wrote:
>> On 10/21/2018 12:20 PM, Nicholas Wilson wrote:
>>> Yes, but the problem you describe is arises from implicit conversion in the other direction, which is not part of the proposal.
>> 
>> It's Manu's example.
>
> Then I don't know what the proposal is. Pieces of it appear to be scattered over numerous posts, mixed in with other text, opinions, and handwavy stuff. There's nothing to point to that is "the proposal".

The proposal is:

Implicit conversion _to_ shared, e.g. passing it to a thread entry point, and not implicit conversion _from_ shared (just like implicit const conversions).

Shared disables reads and writes

 Your confusion results from the use of atomic add which, in Manu's examples had a different signature than before.

> I suggest you and Manu write up a proper proposal. Something that is complete, has nothing else in it, has a rationale, illuminating examples, and explains why alternatives are inferior.

We will eventually. This started as a "please point out any problems with this" and has probably outlived that phase.

> Trying to rewrite the semantics of shared is not a simple task,

Not as much as trying to explain it! Having talked to Manu in person it is much easier to understand.

> doing multithreading correctly is a minefield of "OOPS! I didn't think of that!"

The above case in point, this is about assuming your implementation of thread safe primitives are thread safe (@trusted) that you use it correctly (@safe).

On Sunday, 21 October 2018 at 22:12:18 UTC, Neia Neutuladh wrote:
> On Sun, 21 Oct 2018 12:04:16 -0700, Manu wrote:
>> On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>> Note that there may well be a good way to get the good properties of MP without breaking the type system, but MP itself is not good because it breaks @safe.
>> 
>> Show me. Nobody has been able to show that yet. I'd really like to know this.
>
> If we only used your proposal and only used @safe code, we wouldn't have any data races,

Only of the @trusted implementation is thread safe, which it _should_ be. This is the same caveat as non threaded -@safe/@tusted code.

> but that's only because we wouldn't have any shared data. We'd have shared *variables*, but they would not contain any data we could read

Reads must be atomic.

> or alter, and that's pretty much useless.
>
> To use your proposal, we need to cast data back from shared to unshared.

Yes but this is in the @trusted implementation that forms the basis of your threadsafety.

> When it's unshared, we need to make sure that exactly one thread has a reference to that data as unshared.

Nod.

>And @safe *should* help us with that.

Nod.

> Currently, it helps because casting unshared to shared is not @safe,

This remains the case, and should be done (enforced by the compiler) only in @trusted/@system code as a basis for thread safe, @safe code.

> because it makes it trivial to get multiple threads with unshared references to the same data.

That is @trusted or @system code and therefore is the programmers responsibility.

> And that's when you're using shared as expected rather than doing something weird.

That forms the basis of your thread safe stack. From there on, the basis that shared arguments to functions are treated safely in the presence of threading means that code that calls the @trusted implementations is @safe.

On Sun, Oct 21, 2018 at 11:31 AM Manu <turkeyman@gmail.com> wrote:
>
> On Sun., 21 Oct. 2018, 2:55 am Walter Bright via Digitalmars-d, <digitalmars-d@puremagic.com> wrote:
> >
> > On 10/20/2018 11:24 AM, Manu wrote:
> > > This is an unfair dismissal.
> >
> > It has nothing at all to do with fairness. It is about what the type system guarantees in @safe code. To repeat, the current type system guarantees in @safe code that T* and shared(T)* do not point to the same memory location.
> >
> > Does your proposal maintain that or not? It's a binary question.
>
> By the definition Nick pulled from Wikipedia and posted for you a few posts back, yes, my proposal satisfies Wikipedia's definition of no aliasing. I understand that property is critical, and I have carefully designed for it.
>
> > > I'm not sure you've understood the proposal.
> > > This is the reason for the implicit conversion. It provides safe
> > > transition.
> >
> > I don't see any way to make an implicit T* to shared(T)* safe, or vice versa. The T* code can create more aliases that the conversion doesn't know about, and the shared(T)* code can hand out aliases to other threads. So it all falls to pieces.
>
> T* can't make additional T* aliases on other threads; there can only
> be one thread with T*.
> shared(T)* can not make a T*.
> shared(T)* has no read or write access, so it's not an alias of T* by
> Wikipedia's definition.
>
> Only threadsafe functions can do anything to T.
> The leap of faith is; some @trusted utility functions at the bottom of
> the shared stack makes a promise that it is threadsafe, and must
> deliver that promise.
> I don't think this is unreasonable; this is the nature of @trusted
> functions, they make a promise, and they must keep it.
> If the trusted function does not lie, then the chain of trust holds
> upwards through the stack.
>
> The are very few such trusted functions in practise. Like, similar to the number of digits you have.
>
> > Using a 'scope' qualifier won't work, because 'scope' isn't transitive, while shared is, i.e. U** and shared(U*)*.
>
> I don't think I depend on scope in any way.
> That was an earlier revision of thinking in an older thread.
>
> >  > I'm not sure how to clarify it, what can I give you?
> >
> > Write a piece of code that does such an implicit conversion that you argue is @safe. Make the code as small as possible. Your example:
> >
> >  > int* a;
> >  > shared(int)* b = a;
> >
> > This is not safe.
> >
> > ---- Manu's Proposal ---
> > @safe:
> > int i;
> > int* a = &i;
> > StartNewThread(a); // Compiles! Coder has no idea!
> >
> > ... in the new thread ...
> > void StartOfNewThread(shared(int)* b) {
> >
> >      ... we have two threads accessing 'i',
> >      one thinks it is shared, the other unshared,
> >      and StartOfNewThread() has no idea and anyone
> >      writing code for StartOfNewThread() has no way
> >      to know anything is wrong ...
> >
> >      lockedIncrement(b);  // Data Race!
> > }
>
> This program doesn't compile. You receive an error because it is not safe.
> The function is `lockedIncrement(int*)`. It can't receive a shared
> argument; the function is not threadsafe by my definition.
> You have written a program that produces the expected error that
> alerts you that you have tried to do un-@safe and make a race.
>
> Stanislav produced this same program, and I responded with the correct
> program a few posts back.
> I'll repeat it here; the @safe program to model this interaction is:
>
> @safe:
>
> // function is NOT threadsafe by my definition, can not be called on
> shared arguments
> void atomicIncrement(int*);
>
> struct Atomic(T)
> {
>   // encapsulare the unsafe data so it's inaccessible by any unsafe means
>   private T val;
>
>   // perform the unsafe cast in a trusted function
>   // we are able to assure a valid calling context by encapsulating
> the data above
>   void opUnary(string op : "++")() shared @trusted {
> atomicIncrement(cast(T*)&val); }
> }
>
> Atomic!int i;
> Atomic!int* a = &i;
> StartNewThread(a); // Compiles, of course!
> ++i; // no race
>
> ... in the new thread ...
> void StartOfNewThread(shared(Atomic!int)* b) {
>   ... we have two threads accessing 'i', one has thread-local access,
> this one has a restricted shared access.
>   here, we have a shared instance, so we can only access `b` via
> threadsafe functions.
>   as such, we can manipulate `b` without fear.
>   ++i; // no race!
> }
>
>
> > Your proposal means that the person writing the lockedIncrement(), which is a perfectly reasonable thing to do, simply cannot write it in a way that has a @safe interface
>
> Correct, the rules of my proposal apply to lockedIncrement(). They
> apply to `shared` generally.
> lockedIncrement() is not a threadsafe function. You can't call it on a
> shared instance, because `int`s API (ie, all intrinsic operations) are
> not threadsafe.
> lockedIncrement() can't promise threadsafe access to `shared(int)*`,
> so the argument is not shared.
>
> Your program made the correct compile error about doing unsafety, but
> the location of the compile error is different under my proposal;
> complexity is worn by the shared library author, rather than every
> calling user ever.
> I think my proposal places the complexity in the right location.
> `shared` is intrinsically dangerous; it's not reasonable to ask every
> user that ever calls a shared API to write unsafe code when when
> calling. That's just plain bad design.
>
> > because the person writing the lockedIncrement() library
> > function has no way to know that the data it receives is actually unshared data.
>
> The author of `shared` tooling must assure a valid context such that
> its threadsafety promises are true. Atomic(T) does that with a private
> member.
> The @safe way to interact with atomics is to use the Atomic utility
> type I showed above.
> That is one such @trusted tool that I talk about as being "at the
> bottom of the stack".
> It is probably joined by a mutex/semaphore, and some
> containers/queues. That is probably all the things, and other things
> would be @safe compositions of those tools.
>
> > I.e. @trusted code is obliged to proved a safe interface. Your proposal makes that impossible because the compiler would allow unshared data to be implicitly typed as shared.
>
> What? No.
> Please, try and understand my proposal...


Did you read this email? It seems you didn't read this email... Please read it.

On Sun, Oct 21, 2018 at 2:35 PM Walter Bright via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 10/21/2018 2:08 PM, Walter Bright wrote:
> > On 10/21/2018 12:20 PM, Nicholas Wilson wrote:
> >> Yes, but the problem you describe is arises from implicit conversion in the other direction, which is not part of the proposal.
> >
> > It's Manu's example.
>
> Then I don't know what the proposal is. Pieces of it appear to be scattered over numerous posts, mixed in with other text,

No no, they're repeated, not scattered, because I seem to have to keep repeating it over and over, because nobody is reading the text, or perhaps imaging there is a lot more text than there is.

> opinions, and handwavy stuff.

You mean like every post in opposition which disregards the rules and baselessly asserts it's a terrible idea? :/

> There's nothing to point to that is "the proposal".

You can go back to the OP, not a single detail is changed at any point, but I've repeated it a whole bunch of times (including in direct response to your last post) and the summary has become more concise, but not different.

1. Shared has no read or write access to data
2. Functions with shared arguments are threadsafe with respect to
those arguments
  a. This is a commitment that must be true in _absolute terms_ (there
exists discussion about the ways that neighbours must not undermine
this promise)
  b. There are numerous examples demonstrating how to configure this
(TL;DR: use encapsulation, and @trusted at the bottom of the stack)

If you can find a legitimate example where those rules don't hold, I
want to see it.
But every example so far has been based on a faulty premise where
those 2 simple rules were not actually applied.

I responded to your faulty program directly with the correct program, and you haven't acknowledged it. Did you see it?

> I suggest you and Manu write up a proper proposal. Something that is complete, has nothing else in it, has a rationale, illuminating examples, and explains why alternatives are inferior.

I have written this program a couple of times, including in direct
response to your last sample program.
You seem to have dismissed it... where is your response to that
program, or my last entire post?

> For examples of how to do it:
>
> https://github.com/dlang/DIPs/tree/master/DIPs
>
> Trying to rewrite the semantics of shared is not a simple task, doing multithreading correctly is a minefield of "OOPS! I didn't think of that!" and if anything cries out for a DIP, your and Manu's proposal does.

Yes, I agree it's DIP worthy. But given the almost nothing but overt
hostility I've received here, why on earth would I waste my time
writing a DIP?
I put months into my other DIP which sits gathering dust... if this
thread inspired any confidence that it would be well-received I would
make the effort, but the critical reception we've seen here is... a
bit strange.
It's a 2-point proposal, the rules are **SO SIMPLE**, which is why I
love it. How it can be misunderstood is something I'm having trouble
understanding, and I don't know how to make it any clearer than I
already have; numerous times over, including in my last reply to you,
which you have ignored and dismissed it seems.

Please go back and read my response to your last program.

On 21.10.18 20:46, Manu wrote:
>> Shared data is only useful if, at some point, it is read/written, presumably by
>> casting it to unshared in @trusted code. As soon as that is done, you've got a
>> data race with the other existing unshared aliases.
> If such a race is possible, then the @trusted function is not
> threadsafe, so it is not @trusted by definition.
> You wrote a bad @trusted function, and you should feel bad.
> ...

I wonder where this "each piece of code is maintained by only one person and furthermore this is the only person that will suffer if the code has bugs" mentality comes from. It is very popular as well as obviously nonsense.

> The simplest way to guarantee that no unsafe access is possible is to
> use encapsulation to assure no unregulated access exists.

This only works if untrusted programmers (i.e. programmers who are only allowed to write/modify @safe code) are not allowed to change your class. I.e. it does not work.

On Sun, Oct 21, 2018 at 3:15 PM Neia Neutuladh via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On Sun, 21 Oct 2018 12:04:16 -0700, Manu wrote:
> > On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >> Note that there may well be a good way to get the good properties of MP without breaking the type system, but MP itself is not good because it breaks @safe.
> >
> > Show me. Nobody has been able to show that yet. I'd really like to know this.
>
> If we only used your proposal and only used @safe code, we wouldn't have any data races, but that's only because we wouldn't have any shared data. We'd have shared *variables*, but they would not contain any data we could read or alter, and that's pretty much useless.

I've shown an implementation of Atomic(T) 3 times now... no response any time. Why is it being dismissed? Do I need to write it more times? This is a clear demonstration of how to build the foundation of the @safe threadsafe stack.

> To use your proposal, we need to cast data back from shared to unshared. When it's unshared, we need to make sure that exactly one thread has a reference to that data as unshared.

No, you just need to make sure that access is atomic or synchronised
in a proper way, and if you do cast away shared in some low-level
@trusted function, make sure that reference doesn't escape.
You can do it, I have faith.

> And @safe *should* help us with that.

Totally.

> Currently, it helps because casting unshared to shared is not @safe, because it makes it trivial to get multiple threads with unshared references to the same data.

No no, that's a massive smell. That means anytime anyone wants to
distribute something, they need to perform unsafe casts. That's not
okay.
Modeling shared-ness/unshared-ness is not *useful* in any way that I
have been able to identify. Modelling what it means to be threadsafe
is useful in every application I've ever written.
100% of my SMP code works with my proposal, and something close to 0%
works with shared as it is today. (assuming we desire @safe
interaction, which we do, because threading is hard enough already!)

> And that's when you're using shared as
> expected rather than doing something weird.

No, I *expect* to use shared in @safe code, and not write any unsafe code ever. shared doesn't model a useful interaction now, not in any way.

Today, access to shared data members are unrestricted and completely unsafe, passing data into something like a parallel-for requires unsafe casts.

On 21.10.18 21:04, Manu wrote:
> On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d
> <digitalmars-d@puremagic.com> wrote:
>>
>> On 21.10.18 17:54, Nicholas Wilson wrote:
>>>
>>>> As soon as that is done, you've got a data race with the other
>>>> existing unshared aliases.
>>>
>>> You're in @trusted code, that is the whole point. The onus is on the
>>> programmer to make that correct, same with regular @safe/@trusted@system
>>> code.
>>
>> Not all of the parties that participate in the data race are in @trusted
>> code. The point of @trusted is modularity: you manually check @trusted
>> code according to some set of restrictions and then you are sure that
>> there is no memory corruption.
>>
>> Note that you are not allowed to look at any of the @safe code while
>> checking your @trusted code. You will only see an opaque interface to
>> the @safe code that you call and all you know is that all the @safe code
>> type checks according to @safe rules. Note that there might be an
>> arbitrary number of @safe functions and methods that you do not see.
>>
>> Think about it this way: you first write all the @trusted and @system
>> code, and some evil guy who does not like you comes in after you looks
>> at your code and writes all the @safe code. If there is any memory
>> corruption, it will be your fault and you will face harsh consequences.
>> Now, design the @safe type checking rules. It won't be MP!
>>
>> Note that there may well be a good way to get the good properties of MP
>> without breaking the type system, but MP itself is not good because it
>> breaks @safe.
> 
> Show me. Nobody has been able to show that yet. I'd really like to know this.
> 

I just did, but if you really need to, give me a non-trivial piece of correct multithreaded code that accesses some declared-unshared field from a shared method and I will show you how the evil guy would modify some @safe code in it and introduce race conditions. It needs to be your code, as otherwise you will just claim again that it is me who wrote bad @trusted code.

On Monday, 22 October 2018 at 00:32:35 UTC, Timon Gehr wrote:
> This only works if untrusted programmers (i.e. programmers who are only allowed to write/modify @safe code) are not allowed to change your class. I.e. it does not work.

This is the basis of the current @safe/@trusted/@system model.
Are you saying it is useless?