On Monday, 22 October 2018 at 00:22:19 UTC, Manu wrote:
> On Sun, Oct 21, 2018 at 2:35 PM Walter Bright via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>
>> Then I don't know what the proposal is. Pieces of it appear to be scattered over numerous posts, mixed in with other text,
>
> No no, they're repeated, not scattered, because I seem to have to keep repeating it over and over, because nobody is reading the text, or perhaps imaging there is a lot more text than there is.

I've read every post on this thread and I also have the feeling that it's scattered. At the very least, I'm 90% confident I don't understand what it is you're proposing. Trust me, I'm trying.

I believe that you have a proposal which you believe results in @safe multithreaded code. I don't understand how what I've read so far would accomplish that. I'm conviced that shared data shouldn't be allowed to be written to, but I haven't yet been convinced of anything else.

I don't see how it's possible that implicit conversion from non-shared to shared can work at all. Yes, I know that in the proposal putting `shared` on anything makes it useless, but *somehow* that data gets to be used, even if it's by a @trusted function that casts away shared. At that point, nothing you do thread-safely to the shared data matters if you obtained the shared data from an implicit conversion. There may be many many aliases to it before it was converted, all of them able to write to that memory location believing it's not shared. And it would be @safe (but definitely not thread-safe) to do so! This has been explained a few times, by multiple people. I haven't seen anyone addressing this yet (it's possible it got lost in a sea of text).

I don't even understand why it is you want to cast anything to shared anyway - that'd always be a code smell if I saw it during code review. If it's shared, type it as such. Or better yet, if you can just use immutable.

I understand the frustration of not getting your point across. I would like to kindly point out that, if the replies have gotten to multiple dozen pages and several well-meaning people still don't get it, then the proposal probably isn't as simple as you believe it to be.

On 22.10.18 16:09, Simen Kjærås wrote:
> On Monday, 22 October 2018 at 13:40:39 UTC, Timon Gehr wrote:
>> module reborked;
>> import atomic;
>>
>> void main()@safe{
>>     auto a=new Atomic!int;
>>     import std.concurrency;
>>     spawn((shared(Atomic!int)* a){ ++*a; }, a);
>>     ++a.tupleof[0];
>> }
> 
> Finally! Proof that MP is impossible. On the other hand, why the hell is that @safe? It breaks all sorts of guarantees about @safety. At a minimum, that should be un-@safe.
> 
> Filed in bugzilla: https://issues.dlang.org/show_bug.cgi?id=19326
> 
> -- 
>    Simen

Even if this is changed (and it probably should be), it does not fix the case where the @safe function is in the same module. I don't think it is desirable to change the definition of @trusted such that you need to check the entire module if it contains a single @trusted function.

If I can break safety of some (previously correct) code by editing only @safe code, then that's a significant blow to @safe. I think we need a general way to protect data from being manipulated in @safe code in any way, same module or not.

On Monday, 22 October 2018 at 00:22:19 UTC, Manu wrote:
> On Sun, Oct 21, 2018 at 2:35 PM Walter Bright via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>
>> On 10/21/2018 2:08 PM, Walter Bright wrote:
>> > On 10/21/2018 12:20 PM, Nicholas Wilson wrote:
>> >> Yes, but the problem you describe is arises from implicit conversion in the other direction, which is not part of the proposal.
>> >
>> > It's Manu's example.
>>
>> Then I don't know what the proposal is. Pieces of it appear to be scattered over numerous posts, mixed in with other text,
>
> No no, they're repeated, not scattered, because I seem to have to keep repeating it over and over, because nobody is reading the text, or perhaps imaging there is a lot more text than there is.

I can go look at the original post - and I have - but while it may strictly speaking contain all the information I need, the amount of time and brain power it would take for me to comprehend the consequences is pretty large. I assume you have done a lot of that work already and could save everyone a lot of time by putting together a quick document that covers some of that, with good examples. I'm not going to read {1,2,3}00 messages full of irritated bidirectional miscommunication to try and understand this unless I really have to, and I assume others feel similarly.

On Monday, 22 October 2018 at 14:31:28 UTC, Timon Gehr wrote:
> On 22.10.18 16:09, Simen Kjærås wrote:
>> On Monday, 22 October 2018 at 13:40:39 UTC, Timon Gehr wrote:
>>> module reborked;
>>> import atomic;
>>>
>>> void main()@safe{
>>>     auto a=new Atomic!int;
>>>     import std.concurrency;
>>>     spawn((shared(Atomic!int)* a){ ++*a; }, a);
>>>     ++a.tupleof[0];
>>> }
>> 
>> Finally! Proof that MP is impossible. On the other hand, why the hell is that @safe? It breaks all sorts of guarantees about @safety. At a minimum, that should be un-@safe.
>> 
>> Filed in bugzilla: https://issues.dlang.org/show_bug.cgi?id=19326
>> 
>> --
>>    Simen
>
> Even if this is changed (and it probably should be), it does not fix the case where the @safe function is in the same module. I don't think it is desirable to change the definition of @trusted such that you need to check the entire module if it contains a single @trusted function.
>
> If I can break safety of some (previously correct) code by editing only @safe code, then that's a significant blow to @safe. I think we need a general way to protect data from being manipulated in @safe code in any way, same module or not.

What do you mean by 'previously correct'?

struct Array(T) {
    @safe:
    private int* ptr;
    private int length;
    @disable this();
    this(int n) @trusted {
        ptr = new int[n].ptr;
        length = n;
        foreach (ref e; ptr[0..length])
            e = 123;
    }
    @trusted ref int get(int idx) {
        assert(idx < length);
        return ptr[idx];
    }
}

unittest {
    auto s = Array!int(1);
    assert(s.get(0) == 123);
}

Is this correct code?

What if I add this:

@safe void bork(T)(ref Array!T s) {
    s.length *= 2;
}

unittest {
    auto s = Array!int(1);
    bork(s);
    assert(s.get(1) == 123); // Out of bounds!
}

--
  Simen

On Mon, Oct 22, 2018 at 3:30 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 22.10.18 02:54, Manu wrote:
> > On Sun, Oct 21, 2018 at 5:40 PM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>
> >> On 21.10.18 21:04, Manu wrote:
> >>> On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>>>
> >>>> On 21.10.18 17:54, Nicholas Wilson wrote:
> >>>>>
> >>>>>> As soon as that is done, you've got a data race with the other existing unshared aliases.
> >>>>>
> >>>>> You're in @trusted code, that is the whole point. The onus is on the programmer to make that correct, same with regular @safe/@trusted@system code.
> >>>>
> >>>> Not all of the parties that participate in the data race are in @trusted code. The point of @trusted is modularity: you manually check @trusted code according to some set of restrictions and then you are sure that there is no memory corruption.
> >>>>
> >>>> Note that you are not allowed to look at any of the @safe code while checking your @trusted code. You will only see an opaque interface to the @safe code that you call and all you know is that all the @safe code type checks according to @safe rules. Note that there might be an arbitrary number of @safe functions and methods that you do not see.
> >>>>
> >>>> Think about it this way: you first write all the @trusted and @system code, and some evil guy who does not like you comes in after you looks at your code and writes all the @safe code. If there is any memory corruption, it will be your fault and you will face harsh consequences. Now, design the @safe type checking rules. It won't be MP!
> >>>>
> >>>> Note that there may well be a good way to get the good properties of MP without breaking the type system, but MP itself is not good because it breaks @safe.
> >>>
> >>> Show me. Nobody has been able to show that yet. I'd really like to know this.
> >>>
> >>
> >> I just did,
> >
> > There's no code there... just a presumption that the person who wrote
> > the @trusted code did not deliver the promise they made.
> > ...
>
> Yes, because there is no way to write @trusted code that holds its promise while actually doing something interesting in multiple threads if @safe code can implicitly convert from unshared to shared.

How do my examples prior fail to hold their promise?
struct S
{
  private int x;
  void method() shared @trusted { /* safely manipulate x */ }
}

How can you not trust that function? How can a 3rd party invalidate
that functions promise? `x` is inaccessible.
S can be thread-local or shared as much as you like, and it's safe,
and I don't know how a 3rd party could @safely undermine that?

> >> but if you really need to, give me a non-trivial piece of> correct multithreaded code that accesses some declared-unshared field
> >> from a shared method and I will show you how the evil guy would modify
> >> some @safe code in it and introduce race conditions. It needs to be your
> >> code, as otherwise you will just claim again that it is me who wrote bad
> >> @trusted code.
> >
> > You can pick on any of my prior code fragments. They've all been ignored so far.
> >
>
> I don't want "code fragments". Show me the real code.
>
> I manually browsed through posts now (thanks a lot) and found this
> implementation:
>
> struct Atomic(T){
>    void opUnary(string op : "++")() shared { atomicIncrement(&val); }
>    private T val;
> }
>
> This is @system code. There is no @safe or @trusted here, so I am ignoring it.
>
>
> Then I browsed some more, because I had nothing better to do, and I found this. I completed it so that it is actually compilable, except for the unsafe implicit conversion.
>
> Please read this code, and then carefully read the comments below it before you respond. I will totally ignore any of your answers that arrive in the next two hours.
>
> ---
> module borked;
>
> void atomicIncrement(int* p)@system{
>      import core.atomic;
>      atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
> }
>
> struct Atomic(T){
>      private T val;
>      void opUnary(string op : "++")() shared @trusted {
>          atomicIncrement(cast(T*)&val);
>      }
> }
> void main()@safe{
>      Atomic!int i;
>      auto a=&[i][0];// was: Atomic!int* a = &i;
>      import std.concurrency;
>      spawn((shared(Atomic!int)* a){ ++*a; }, a);
>      ++i.val; // race
> }
> ---
>
>
> Oh no! The author of the @trusted function (i.e. you) did not deliver on
> the promise they made!
>
> Now, before you go and tell me that I am stupid because I wrote bad code, consider the following:
>
> - It is perfectly @safe to access private members from the same module.
>
> - You may not blame the my @safe main function for the problem. It is @safe, so it cannot be blamed for UB. Any UB is the result of a bad @trusted function, a compiler bug, or hardware failure.
>
> - The only @trusted function in this module was written by you.
>
> You said that there is a third implementation somewhere. If that one actually works, I apologize and ask you to please paste it again in this subthread.

Last time I checked, main does not go in core.atomic. I've never added
any code to core.atomic.
These things wouldn't live in peoples code.

I understand that my proposal relies on trusting a very small number
of low-level implementations at the bottom of the stack... but they're
strongly encapsulated, live in libraries, and if there existed a world
where you COULD describe threadsafe interaction with shared; you
would.
@trusted user code, especially where it related to `shared` would be
terrifying, and you wouldn't do it.
I understand that the situation you present is technically possible,
but why would it happen in reality? I find the proposition of a @safe
threading stack to far far outweight that risk.
It's also possible that tech may be improved to assist with this
particular problem, I haven't tried to address it; I'm interested in
if the rules are sound.

One suggestion was to make `private int val` shared, then external functions have no access... that helps mitigate this particular mistake.

It's back to this 1:many thing. There's one place you can possibly
make this mistake (and it's probably maintained by an expert author),
and it should be well encapsulated in a core lib; whereas the current
`shared` requires that _end-users_ do unsafe casts all the time among
user code, and they're almost certainly not experts.
That's rigged totally backwards. The values are all wrong.

If my scheme is sound above this issue, then it's reasonable to focus on ideas to reduce the odds of mistake for the one implementer of the low-level tool.

On Mon, Oct 22, 2018 at 4:50 AM Stanislav Blinov via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On Monday, 22 October 2018 at 00:22:19 UTC, Manu wrote:
>
> > No no, they're repeated, not scattered, because I seem to have
> > to keep repeating it over and over, because nobody is reading
> > the text, or perhaps imaging there is a lot more text than
> > there is.
> > ...
> > You mean like every post in opposition which disregards the
> > rules and baselessly asserts it's a terrible idea? :/
> > ...
> > I responded to your faulty program directly with the correct
> > program, and you haven't acknowledged it. Did you see it?
>
> Right back at you.
>
> Quote:
>
> I think this is a typical sort of construction:
>
> struct ThreadsafeQueue(T)
> {
>    void QueueItem(T*) shared;
>    T* UnqueueItem() shared;
> }
>
> struct SpecialWorkList
> {
>    struct Job { ... }
>
>    void MakeJob(int x, float y, string z) shared  // <- any thread
> may
> produce a job
>    {
>      Job* job = new Job; // <- this is thread-local
>      PopulateJob(job, x, y, z); // <- preparation of a job might be
> complex, and worthy of the SpecialWorkList implementation
>
>      jobList.QueueItem(job);  // <- QueueItem encapsulates
> thread-safety, no need for blunt casts
>    }
>
>    void Flush() // <- not shared, thread-local consumer
>    {
>      Job* job;
>      while (job = jobList.UnqueueItem()) // <- it's obviously safe
> for
> a thread-local to call UnqueueItem even though the implementation
> is
> threadsafe
>      {
>        // thread-local dispatch of work...
>        // perhaps rendering, perhaps deferred destruction, perhaps
> deferred resource creation... whatever!
>      }
>    }
>
>    void GetSpecialSystemState() // <- this has NOTHING to do with
> the
> threadsafe part of SpecialWorkList
>    {
>      return os.functionThatChecksSystemState();
>    }
>
>    // there may be any number of utility functions that don't
> interact
> with jobList.
>
> private:
>    void PopulateJob(ref Job job, ...)
>    {
>      // expensive function; not thread-safe, and doesn't have any
> interaction with threading.
>    }
>
>    ThreadsafeQueue!Job jobList;
> }
>
>
> This isn't an amazing example, but it's typical of a thing that's
> mostly thread-local, and only a small controlled part of it's
> functionality is thread-safe.
> The thread-local method Flush() also deals with thread-safety
> internally... because it flushes a thread-safe queue.
>
> All thread-safety concerns are composed by a utility object, so there's no need for locks, magic, or casts here.
>
> EndQuote;
>
> The above:
> 1) Will not compile, not currently, not under your proposal
> (presumably you forgot in frustration to cast before calling
> PopulateJob?..)

I did correct that line (along with an apology) on my very next post; it would probably be a member of Job... or any manner of other code. That is the least interesting line in the program

> 2) Does not in any way demonstrate a practical @safe application of an implicit conversion. As I wrote in the original response to that code, with that particular code it seems more like you just need forwarding methods that call `shared` methods under the hood (i.e. MakeJob), and it'd be "nice" if you didn't have to write those and could just call `shared` MakeJob on an un-`shared` reference directly. But these are all assumptions without seeing the actual usage.
>
> Please just stop acting like everyone here is opposing *you*.

You're right, it's mostly you.

>  All
> you're doing is dismissing everyone with a "nuh-huh, you no
> understand, you bad". If it was just me, fine, it would mean I'm
> dumb and not worthy of this discussion. But this isn't the case,
> which means *you are not getting your point across*. And yet
> instead of trying to fix that, you're getting all snarky.

I mean, it's fair, but it's pretty bloody hypocritical coming from you!
I think it's fair to point out that your high-frequency, persistent,
and unwavering hostility from post #1 across all my recent threads (at
least, until I told you to GF) is the primary reason I'm frustrated
here.
You can own part responsibility for my emotion.

On Mon, Oct 22, 2018 at 6:00 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 22.10.18 12:26, Timon Gehr wrote:
> > ---
> > module borked;
> >
> > void atomicIncrement(int* p)@system{
> >      import core.atomic;
> >      atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
> > }
> >
> > struct Atomic(T){
> >      private T val;
> >      void opUnary(string op : "++")() shared @trusted {
> >          atomicIncrement(cast(T*)&val);
> >      }
> > }
> > void main()@safe{
> >      Atomic!int i;
> >      auto a=&[i][0];// was: Atomic!int* a = &i;
> >      import std.concurrency;
> >      spawn((shared(Atomic!int)* a){ ++*a; }, a);
> >      ++i.val; // race
> > }
> > ---
>
> Obviously, this should have been:
>
> ---
> module borked;
>
> void atomicIncrement(int*p)@system{
>      import core.atomic;
>      atomicOp!"+="(*cast(shared(int)*)p,1);
> }
> struct Atomic(T){
>      private T val;
>      void opUnary(string op:"++")()shared @trusted{
>          atomicIncrement(cast(T*)&val);
>      }
> }
> void main()@safe{
>      auto a=new Atomic!int;
>      import std.concurrency;
>      spawn((shared(Atomic!int)* a){ ++*a; }, a);
>      ++a.val; // race
> }
> ---
>
> (I was short on time and had to fix Manu's code because it was not
> actually compilable.)

Nitpick; atomicOp does not receive a shared arg under my proposal, it's not a threadsafe function by definition as discussed a few times.

On Mon, Oct 22, 2018 at 6:40 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 22.10.18 03:01, Manu wrote:
> > Where did I ever say anything like that? I'm sure I've never said this.
>
> ???
>
> I said that you are proposing to allow implicit conversions to shared for all classes, not only core.atomic.Atomic, and the last time you said it was the previous sentence of the same post.

Sorry, I read the "not proposing to let" as something like "proposing
to not let".
So let me re-respond.

Yes, I propose implicit conversion to shared. That's what makes it usable.
Basically any argument to a fork/join, parallel-for, map/reduce,
etc... they all require this transition.
Our software is almost exclusively made up of those processes. Almost
every type we have transits to shared contexts (with a restricted
threadsafe API), and almost nothing we have could be allocated shared
to the exclusion of thread-local access (like Atila likes to suggest).
I don't know how to make use of the mutually-exclusive design of the
current model in any code I've ever written.

I've been writing SMP code since the xbox360 alpha-kit landed on my
desk in 2006. As we've matured, we've seen mutexes disappear
completely. Even discreet worker threads which used to use semaphores
to signal new work arrival are dispappearing as it's becoming
impossible to find enough distinct kinds of work for a discreet worker
threads to do that keeps all cores busy.
We break the work into tasks, build a DAG of the execution schedule
with respect to data access dependencies, and parallel-for/reduce
within tasks that operate over large volumes of data.
We used to hide data sharing detail behind walls, but it moves into
user-facing code as parallel-for appears, and that invokes the
necessity to be able to express what is threadsafe at the type system
level.

I can assure, in our architecture at least, I am not aware of any case where a user would write a @trusted function under my proposal. We could run our stack @safe.

> >>> You seem to be stuck on the detail whether you can trust the @trusted author though...
> >>
> >> Again: the @safe author is the problem.
> >
> > I don't follow. The @safe author is incapable of doing threadsafety violation.
>
> They are capable of doing so as soon as you provide them a @trusted function that treats data as shared that they can access as unshared.

That function is not @trusted by definition.

> > They can only combine threadsafe functions.
> > They can certainly produce a program that doesn't work, and they are
> > capable of ordering issues, but that's not the same as data-race
> > related crash bugs.
>
> Accessing private members of aggregates in the same module is @safe. tupleof is @safe too.

I don't see any situation where any user would write code in the same module as core.atomic, or core.mutex, or wherever these couple of functions live.

On Mon, Oct 22, 2018 at 7:10 AM Simen Kjærås via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On Monday, 22 October 2018 at 13:40:39 UTC, Timon Gehr wrote:
> > module reborked;
> > import atomic;
> >
> > void main()@safe{
> >     auto a=new Atomic!int;
> >     import std.concurrency;
> >     spawn((shared(Atomic!int)* a){ ++*a; }, a);
> >     ++a.tupleof[0];
> > }
>
> Finally! Proof that MP is impossible. On the other hand, why the hell is that @safe? It breaks all sorts of guarantees about @safety. At a minimum, that should be un-@safe.
>
> Filed in bugzilla: https://issues.dlang.org/show_bug.cgi?id=19326

Yeah, that's shockingly dangerous for all sorts of reasons!
I mean, is this really an argument to destroy my proposal, or are you
just destroying @safe in general?

On Mon, Oct 22, 2018 at 7:35 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 22.10.18 16:09, Simen Kjærås wrote:
> > On Monday, 22 October 2018 at 13:40:39 UTC, Timon Gehr wrote:
> >> module reborked;
> >> import atomic;
> >>
> >> void main()@safe{
> >>     auto a=new Atomic!int;
> >>     import std.concurrency;
> >>     spawn((shared(Atomic!int)* a){ ++*a; }, a);
> >>     ++a.tupleof[0];
> >> }
> >
> > Finally! Proof that MP is impossible. On the other hand, why the hell is that @safe? It breaks all sorts of guarantees about @safety. At a minimum, that should be un-@safe.
> >
> > Filed in bugzilla: https://issues.dlang.org/show_bug.cgi?id=19326
> >
> > --
> >    Simen
>
> Even if this is changed (and it probably should be), it does not fix the case where the @safe function is in the same module. I don't think it is desirable to change the definition of @trusted such that you need to check the entire module if it contains a single @trusted function.
>
> If I can break safety of some (previously correct) code by editing only @safe code, then that's a significant blow to @safe. I think we need a general way to protect data from being manipulated in @safe code in any way, same module or not.

I'm all ears.