On Sun, 21 Oct 2018 17:35:38 -0700, Manu wrote:

> On Sun, Oct 21, 2018 at 3:15 PM Neia Neutuladh via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>> If we only used your proposal and only used @safe code, we wouldn't have any data races, but that's only because we wouldn't have any shared data. We'd have shared *variables*, but they would not contain any data we could read or alter, and that's pretty much useless.
> 
> I've shown an implementation of Atomic(T) 3 times now... no response any
> time. Why is it being dismissed? Do I need to write it more times?
> This is a clear demonstration of how to build the foundation of the
> @safe threadsafe stack.

Yes, Atomic can be implemented here as @trusted code, and maybe using @safe compiler intrinsics for some situations. That's useful! But it's not *enough*.

It would require a lot of work to refit existing code to using only atomic operations, and it would end up looking rather clunky a lot of the time. If you're doing anything complex with a complex object graph, you're going to have a terrible time. You need to copy that object graph (atomically), which is going to be expensive and provides non-trivial restrictions on what you can store.

So I'm not keen on a world in which all multithreading is using atomic structs.

Unless, when you were talking about Atomic, you actually meant a wrapper containing some sort of lock, allowing you to submit arbitrary delegates to mutate the data within in a serialized way, similar to Atila Neves's fearless library. Which really stretches the definition of "atomic".

>> Currently, it helps because casting unshared to shared is not @safe, because it makes it trivial to get multiple threads with unshared references to the same data.
> 
> No no, that's a massive smell. That means anytime anyone wants to distribute something, they need to perform unsafe casts. That's not okay.

Casting thread-local to shared makes it easy to cause errors, and that's why it's a massive smell. Making it silent doesn't eliminate the smell.

> 100% of my SMP code works with my proposal, and something close to 0% works with shared as it is today. (assuming we desire @safe interaction, which we do, because threading is hard enough already!)

You want un-shared things to implicitly cast to shared. You don't want to have to allocate anything as shared, and you do want to pass absolutely anything to any thread.

You can write a @trusted assumeShared function, analogous to assumeUnique, to accomplish that. It would be slightly more awkward than what you're proposing, but it accomplishes one of your two goals: convert non-shared things to shared things in @safe code.

The other goal in your proposal is for some code that currently compiles not to. So you should be able to write the code you want, albeit with an increased risk of bugs. Or you could write a template that wraps a shared thing and forbids field access and assignment.

>> And that's when you're using shared as expected rather than doing something weird.
> 
> No, I *expect* to use shared in @safe code, and not write any unsafe code ever. shared doesn't model a useful interaction now, not in any way.
> 
> Today, access to shared data members are unrestricted and completely unsafe

Yes, and that's bad and should be changed.

> passing data into something like a parallel-for requires unsafe casts.

Or allocating data as shared, which is the recommended way, because that makes absolutely certain that, from the start, no code has an un-shared copy of that data. No casts needed.

On Monday, 22 October 2018 at 00:22:19 UTC, Manu wrote:
> On Sun, Oct 21, 2018 at 2:35 PM Walter Bright via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>
>> On 10/21/2018 2:08 PM, Walter Bright wrote:
>> > On 10/21/2018 12:20 PM, Nicholas Wilson wrote:
>> >> Yes, but the problem you describe is arises from implicit conversion in the other direction, which is not part of the proposal.
>> >
>> > It's Manu's example.
>>
>> Then I don't know what the proposal is. Pieces of it appear to be scattered over numerous posts, mixed in with other text,
>
> No no, they're repeated, not scattered, because I seem to have to keep repeating it over and over, because nobody is reading the text, or perhaps imaging there is a lot more text than there is.

I told you this is what happens with forum posts 4 days ago, yet you didn't listen:

https://forum.dlang.org/post/fokdcnzircoiuhrhzmgv@forum.dlang.org

>> opinions, and handwavy stuff.
>
> You mean like every post in opposition which disregards the rules and baselessly asserts it's a terrible idea? :/
>
>> There's nothing to point to that is "the proposal".
>
> You can go back to the OP, not a single detail is changed at any point, but I've repeated it a whole bunch of times (including in direct response to your last post) and the summary has become more concise, but not different.
>
> 1. Shared has no read or write access to data
> 2. Functions with shared arguments are threadsafe with respect to
> those arguments
>   a. This is a commitment that must be true in _absolute terms_ (there
> exists discussion about the ways that neighbours must not undermine
> this promise)
>   b. There are numerous examples demonstrating how to configure this
> (TL;DR: use encapsulation, and @trusted at the bottom of the stack)
>
> If you can find a legitimate example where those rules don't hold, I
> want to see it.
> But every example so far has been based on a faulty premise where
> those 2 simple rules were not actually applied.

Put it all together in a 2-3 page proposal elsewhere, so he doesn't have to hunt everything out in a blizzard of forum posts.

> I responded to your faulty program directly with the correct program, and you haven't acknowledged it. Did you see it?
>
>> I suggest you and Manu write up a proper proposal. Something that is complete, has nothing else in it, has a rationale, illuminating examples, and explains why alternatives are inferior.
>
> I have written this program a couple of times, including in direct
> response to your last sample program.
> You seem to have dismissed it... where is your response to that
> program, or my last entire post?
>
>> For examples of how to do it:
>>
>> https://github.com/dlang/DIPs/tree/master/DIPs
>>
>> Trying to rewrite the semantics of shared is not a simple task, doing multithreading correctly is a minefield of "OOPS! I didn't think of that!" and if anything cries out for a DIP, your and Manu's proposal does.
>
> Yes, I agree it's DIP worthy. But given the almost nothing but overt
> hostility I've received here, why on earth would I waste my time
> writing a DIP?
> I put months into my other DIP which sits gathering dust... if this
> thread inspired any confidence that it would be well-received I would
> make the effort, but the critical reception we've seen here is... a
> bit strange.
> It's a 2-point proposal, the rules are **SO SIMPLE**, which is why I
> love it. How it can be misunderstood is something I'm having trouble
> understanding, and I don't know how to make it any clearer than I
> already have; numerous times over, including in my last reply to you,
> which you have ignored and dismissed it seems.
>
> Please go back and read my response to your last program.

He did not say to write a full DIP, just a proposal, so he knows exactly what you mean, just as I said. It will require a DIP eventually, but he didn't ask you to write one now.

On 10/21/2018 5:54 PM, Manu wrote:
> Would you please respond to my messages, and specifically, respond to
> the code that I presented to you in response to your broken example.
> Or any of my earlier fragments throughout this thread. I've shared
> quite a few, and so far, nobody has ever produced a criticism of any
> of my fragments. They've just been skipped over.

That's just the problem. You've posted 62 messages so far in this thread, and then there's all the ones Nicholas posted.

Trying to assemble the "earlier fragments throughout this thread" is not practical for readers, and the endless nature of this thread is ample evidence for it. The n.g. is a place to discuss a proposal, not the proposal itself.

This change is definitely merits an actual proposal DIP, so that one is assured of seeing the complete proposal, rationale, examples, etc., in one document, as well as not being distracted by sidebars, thread drift, and mistakes. This document can evolve with corrections and clarifications from the discussion, and anyone can get up to speed quickly by just reading the latest version of it.


> But the one aimed directly at your own most recent sample program
> addresses your program directly.

My most recent sample program was a direct criticism of one of your fragments, so please don't say "nobody has ever ...". I do understand your frustration at finding it hard to get your point across, but the problem at least for me is trying to mine it from nuggets scattered across 62 posts. Mine it, refine it, cast it into an ingot, then present it as a DIP.

On 10/21/2018 11:58 AM, Timon Gehr wrote:
> [...]

Thank you, Timon, for a nice explanation of what I was trying to express.

On Mon, Oct 22, 2018 at 12:50 AM Walter Bright via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 10/21/2018 5:54 PM, Manu wrote:
> > Would you please respond to my messages, and specifically, respond to the code that I presented to you in response to your broken example. Or any of my earlier fragments throughout this thread. I've shared quite a few, and so far, nobody has ever produced a criticism of any of my fragments. They've just been skipped over.
>
> That's just the problem. You've posted 62 messages so far in this thread, and then there's all the ones Nicholas posted.

I sent it twice... again just a short while ago right before this one... but you responded to this one and not that one O_o

> Trying to assemble the "earlier fragments throughout this thread" is not practical for readers, and the endless nature of this thread is ample evidence for it. The n.g. is a place to discuss a proposal, not the proposal itself.
>
> This change is definitely merits an actual proposal DIP, so that one is assured of seeing the complete proposal, rationale, examples, etc., in one document, as well as not being distracted by sidebars, thread drift, and mistakes. This document can evolve with corrections and clarifications from the discussion, and anyone can get up to speed quickly by just reading the latest version of it.

Okay, but I still want you to respond to my corrections of your program, which were in direct response to you... twice.

>  > But the one aimed directly at your own most recent sample program
>  > addresses your program directly.
>
> My most recent sample program was a direct criticism of one of your fragments, so please don't say "nobody has ever ...". I do understand your frustration at finding it hard to get your point across, but the problem at least for me is trying to mine it from nuggets scattered across 62 posts. Mine it, refine it, cast it into an ingot, then present it as a DIP.

I posted it, twice... 2 messages, back to back, and you're responding to this one, and not that one. I'll post it again...

Third time's the charm.... maybe?

--------------------- repeated, 3rd time ------------------------

On Sun., 21 Oct. 2018, 2:55 am Walter Bright via Digitalmars-d, <digitalmars-d@puremagic.com> wrote:
>
> On 10/20/2018 11:24 AM, Manu wrote:
> > This is an unfair dismissal.
>
> It has nothing at all to do with fairness. It is about what the type system guarantees in @safe code. To repeat, the current type system guarantees in @safe code that T* and shared(T)* do not point to the same memory location.
>
> Does your proposal maintain that or not? It's a binary question.

By the definition Nick pulled from Wikipedia and posted for you a few posts back, yes, my proposal satisfies Wikipedia's definition of no aliasing. I understand that property is critical, and I have carefully designed for it.

> > I'm not sure you've understood the proposal.
> > This is the reason for the implicit conversion. It provides safe
> > transition.
>
> I don't see any way to make an implicit T* to shared(T)* safe, or vice versa. The T* code can create more aliases that the conversion doesn't know about, and the shared(T)* code can hand out aliases to other threads. So it all falls to pieces.

T* can't make additional T* aliases on other threads; there can only
be one thread with T*.
shared(T)* can not make a T*.
shared(T)* has no read or write access, so it's not an alias of T* by
Wikipedia's definition.

Only threadsafe functions can do anything to T.
The leap of faith is; some @trusted utility functions at the bottom of
the shared stack makes a promise that it is threadsafe, and must
deliver that promise.
I don't think this is unreasonable; this is the nature of @trusted
functions, they make a promise, and they must keep it.
If the trusted function does not lie, then the chain of trust holds
upwards through the stack.

The are very few such trusted functions in practise. Like, similar to the number of digits you have.

> Using a 'scope' qualifier won't work, because 'scope' isn't transitive, while shared is, i.e. U** and shared(U*)*.

I don't think I depend on scope in any way.
That was an earlier revision of thinking in an older thread.

>  > I'm not sure how to clarify it, what can I give you?
>
> Write a piece of code that does such an implicit conversion that you argue is @safe. Make the code as small as possible. Your example:
>
>  > int* a;
>  > shared(int)* b = a;
>
> This is not safe.
>
> ---- Manu's Proposal ---
> @safe:
> int i;
> int* a = &i;
> StartNewThread(a); // Compiles! Coder has no idea!
>
> ... in the new thread ...
> void StartOfNewThread(shared(int)* b) {
>
>      ... we have two threads accessing 'i',
>      one thinks it is shared, the other unshared,
>      and StartOfNewThread() has no idea and anyone
>      writing code for StartOfNewThread() has no way
>      to know anything is wrong ...
>
>      lockedIncrement(b);  // Data Race!
> }

This program doesn't compile. You receive an error because it is not safe.
The function is `lockedIncrement(int*)`. It can't receive a shared
argument; the function is not threadsafe by my definition.
You have written a program that produces the expected error that
alerts you that you have tried to do un-@safe and make a race.

Stanislav produced this same program, and I responded with the correct
program a few posts back.
I'll repeat it here; the @safe program to model this interaction is:

@safe:

// function is NOT threadsafe by my definition, can not be called on
shared arguments
void atomicIncrement(int*);

struct Atomic(T)
{
  // encapsulare the unsafe data so it's inaccessible by any unsafe means
  private T val;

  // perform the unsafe cast in a trusted function
  // we are able to assure a valid calling context by encapsulating
the data above
  void opUnary(string op : "++")() shared @trusted {
atomicIncrement(cast(T*)&val); }
}

Atomic!int i;
Atomic!int* a = &i;
StartNewThread(a); // Compiles, of course!
++i; // no race

... in the new thread ...
void StartOfNewThread(shared(Atomic!int)* b) {
  //... we have two threads accessing 'i', one has thread-local
access, this one has a restricted shared access.
  // here, we have a shared instance, so we can only access `b` via
threadsafe functions.
  // as such, we can manipulate `b` without fear.
  ++i; // no race!
}


> Your proposal means that the person writing the lockedIncrement(), which is a perfectly reasonable thing to do, simply cannot write it in a way that has a @safe interface

Correct, the rules of my proposal apply to lockedIncrement(). They
apply to `shared` generally.
lockedIncrement() is not a threadsafe function. You can't call it on a
shared instance, because `int`s API (ie, all intrinsic operations) are
not threadsafe.
lockedIncrement() can't promise threadsafe access to `shared(int)*`,
so the argument is not shared.

Your program made the correct compile error about doing unsafety, but
the location of the compile error is different under my proposal;
complexity is worn by the shared library author, rather than every
calling user ever.
I think my proposal places the complexity in the right location.
`shared` is intrinsically dangerous; it's not reasonable to ask every
user that ever calls a shared API to write unsafe code when when
calling. That's just plain bad design.

> because the person writing the lockedIncrement() library
> function has no way to know that the data it receives is actually unshared data.

The author of `shared` tooling must assure a valid context such that
its threadsafety promises are true. Atomic(T) does that with a private
member.
The @safe way to interact with atomics is to use the Atomic utility
type I showed above.
That is one such @trusted tool that I talk about as being "at the
bottom of the stack".
It is probably joined by a mutex/semaphore, and some
containers/queues. That is probably all the things, and other things
would be @safe compositions of those tools.

> I.e. @trusted code is obliged to proved a safe interface. Your proposal makes that impossible because the compiler would allow unshared data to be implicitly typed as shared.

What? No.
Please, try and understand my proposal...

On Mon, Oct 22, 2018 at 12:55 AM Walter Bright via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 10/21/2018 11:58 AM, Timon Gehr wrote:
> > [...]
>
> Thank you, Timon, for a nice explanation of what I was trying to express.

You removed whatever comment you're referring to.
I don't understand any of Timon's posts in this thread at all, which
is unusual because he's usually pretty clear.

On 10/22/2018 1:42 AM, Manu wrote:
> You removed whatever comment you're referring to.

If your newsreader cannot find the antecedent, you badly need to use a better one. Thunderbird handles this rather well, there's no reason to use an inferior one.

Or just click the <- button:

https://digitalmars.com/d/archives/digitalmars/D/shared_-_i_need_it_to_be_useful_320165.html#N320607


> I don't understand any of Timon's posts in this thread at all, which
> is unusual because he's usually pretty clear.

I suspect Timon is equally frustrated at not getting his point across.

On 10/22/2018 1:34 AM, Manu wrote:
> I posted it, twice... 2 messages, back to back, and you're responding
> to this one, and not that one. I'll post it again...


Posting it over and over is illustrative of the failure of posting proposal documents to the n.g. instead of posting it as a DIP which can be referred to:

1. nobody knows which of your 70 messages are the ones with the proposal in it

2. with multiple posts of the proposal, nobody knows which one is the most up-to-date one

Doing it this way does not work. Continuing to repost it is a waste of your time. Post it as a DIP and link to it.

On 22/10/2018 10:28 PM, Walter Bright wrote:
> On 10/22/2018 1:34 AM, Manu wrote:
>> I posted it, twice... 2 messages, back to back, and you're responding
>> to this one, and not that one. I'll post it again...
> 
> 
> Posting it over and over is illustrative of the failure of posting proposal documents to the n.g. instead of posting it as a DIP which can be referred to:
> 
> 1. nobody knows which of your 70 messages are the ones with the proposal in it
> 
> 2. with multiple posts of the proposal, nobody knows which one is the most up-to-date one
> 
> Doing it this way does not work. Continuing to repost it is a waste of your time. Post it as a DIP and link to it.

As I've said previously, it doesn't need to be a good DIP or anywhere near complete. It just needs code examples comparing current and proposed behavior with some text about semantics changes.