On Wed, Oct 17, 2018 at 6:15 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 17.10.2018 14:24, Timon Gehr wrote:
> > and unshared methods are only allowed to access unshared members.
>
> This is actually not necessary, let me reformulate:
>
> You want:
>
> - if you have a C c and a shared(C) s, typeof(s.x) == typeof(c.x).

No. c.x is mutable, s.x is shared (and inaccessible).

> - shared methods are not allowed to access unshared members.

No. Shared methods are not allowed to access ANY members. Shared data
has no access.
I don't believe there's any rules that can make raw access to shared
data safe. We must depend on threadsafe tooling (libraries).

> - shared is not transitive, and therefore unshared class references implicitly convert to shared class references

No. Shared is transitive.

> Applied to pointers, this would mean that you can implicitly convert
> int* -> shared(int*), but not shared(int*)->int*, int* -> shared(int)*
> or shared(int)* -> int*.

Correct. This is acceptable because shared(int)* can not be accessed;
can't produce to a race.
shared -> unshared conversion produces an unshared alias, and is
obviously invalid.

> shared(int*) and shared(shared(int)*) would be
> different types, such that shared(int*) cannot be dereferenced but
> shared(shared(int)*) can.

No. No manner of shared(T) is safely accessible under any
circumstance. This is just the reality, and there's no way it can be
allowed.
The escape hatch is that T may have some shared methods which
implement threadsafe interaction with T, and THAT is the only safe way
to interact with shared data.

On Wed, Oct 17, 2018 at 5:05 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> [... all text ...]

OMFG, I just spent about 3 hours writing a super-detailed reply to all
of Timon's posts in aggregate... I clicked send... and it's gone.
I don't know if this is a gmail thing, a mailing list thing... no
idea... but it's... gone.
I can't repeat that effort :(

On Thursday, 18 October 2018 at 06:20:02 UTC, Manu wrote:
> On Wed, Oct 17, 2018 at 5:05 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>
>> [... all text ...]
>
> OMFG, I just spent about 3 hours writing a super-detailed reply to all
> of Timon's posts in aggregate... I clicked send... and it's gone.
> I don't know if this is a gmail thing, a mailing list thing... no
> idea... but it's... gone.
> I can't repeat that effort :(

Never never write something super-detailed in a web-based "thing"!
Native application, and copy-past!

:-O

But' now I'm curious about your reply! Timon argumentation are really strong (IMHO), so it's a double effort! :-/

/Paolo

Manu,

how is it that you can't see what *your own* proposal means??? Implicit casting from mutable to shared means that everything is shared by default! Precisely the opposite of what D proclaims.

You also essentially forbid defining *any* functions that take `shared T*` argument(s). You keep asking for concrete "holes". Don't you see what the previous "atomicInc" example implies???

If *any* free function `foo(shared T* bar)`, per your definition, is not threadsafe, then no other function with shared argument(s) can be threadsafe at all. So how do you call functions on shared data then? You keep saying "methods, methods..."

struct Other { /* ... */ }

struct S {
    void foo(shared Other*) shared;
}

Per your rules, there would be *nothing* in the language to prevent calling S.foo with an unshared Other.

So the only way to make your proposal work would be to forbid all functions from taking `shared T*` or `ref shared T` argument. Except we can't do that, because a method is just a function with an implicit first argument. The code above is the same as this:

void foo(ref shared S, shared Other*);

It's literally *the same signature*. So there's nothing in the language to prevent calling that on an unshared S either.

To sum up, things you implied but never specified in your proposal:

1. Primitive types can't be explicitly `shared`.
2. Free functions taking `shared` arguments are not allowed.
3. Only `shared` methods can implement threadsafe operations on `shared` data (which contradicts (2) already) <- this one you did specify.
4. Every variable is implicitly shared, whether intended so or not.

On Thursday, 18 October 2018 at 10:08:48 UTC, Stanislav Blinov wrote:
> Manu,
>
> how is it that you can't see what *your own* proposal means??? Implicit casting from mutable to shared means that everything is shared by default! Precisely the opposite of what D proclaims.

Well, sorta. But that's not a problem, because you can't do anything that's not threadsafe to something that's shared.


> You also essentially forbid defining *any* functions that take `shared T*` argument(s). You keep asking for concrete "holes". Don't you see what the previous "atomicInc" example implies???

I certainly don't. Please do elucidate.


> If *any* free function `foo(shared T* bar)`, per your definition, is not threadsafe, then no other function with shared argument(s) can be threadsafe at all. So how do you call functions on shared data then? You keep saying "methods, methods..."
>
> struct Other { /* ... */ }
>
> struct S {
>     void foo(shared Other*) shared;
> }
>
> Per your rules, there would be *nothing* in the language to prevent calling S.foo with an unshared Other.

That's true. And you can't do anything to it, so that's fine.


> So the only way to make your proposal work would be to forbid all functions from taking `shared T*` or `ref shared T` argument.

No. Please read this thread again. From the beginning, every word. Actually, don't do that, because Manu's proposal is simple and elegant:

> 1. the rule must be applied that shared object can not be read or written
> 2. attributing a method shared is a statement and a promise that the
> method is threadsafe
> 
> The rest just follows naturally.

There's actually one more thing: The one and only thing you can do (without unsafe casting) with a shared object, is call shared methods and free functions on it.


> To sum up, things you implied but never specified in your proposal:
>
> 1. Primitive types can't be explicitly `shared`.

Sure they can, they just can't present a thread-safe interface, so you can't do anything with a shared(int).


> 2. Free functions taking `shared` arguments are not allowed.

Yes, they are. They would be using other shared methods or free functions on the shared argument, and would thus be thread-safe. If defined in the same module as the type on which they operate, they would have access to the internal state of the object, and would have to be written in such a way as to not violate the thread-safety of other methods and free functions that operate on it.


> 3. Only `shared` methods can implement threadsafe operations on `shared` data (which contradicts (2) already) <- this one you did specify.

Non-shared methods are perfectly free to be thread-safe (and they should be, in the sense that they shouldn't interfere with shared methods). A better way to state this is that only shared methods may be called on a shared object. A shared object may also be passed to a function taking a shared parameter.


> 4. Every variable is implicitly shared, whether intended so or not.

Well, yes, in the same sense that every variable is also implicitly const, whether intended so or not.

--
  Simen

On Thursday, 18 October 2018 at 11:35:21 UTC, Simen Kjærås wrote:
> On Thursday, 18 October 2018 at 10:08:48 UTC, Stanislav Blinov wrote:
>> Manu,
>>
>> how is it that you can't see what *your own* proposal means??? Implicit casting from mutable to shared means that everything is shared by default! Precisely the opposite of what D proclaims.
>
> Well, sorta. But that's not a problem, because you can't do anything that's not threadsafe to something that's shared.

Yes you can. You silently agree to another function's assumption that you pass shared data, while actually passing thread-local data and keeping treating it as thread-local. I.e. you silently agree to a race.

>> You also essentially forbid defining *any* functions that take `shared T*` argument(s). You keep asking for concrete "holes". Don't you see what the previous "atomicInc" example implies???
>
> I certainly don't. Please do elucidate.

What the hell? I do in the very next paragraph. Do people read sentence by sentence and assume context does not exist or what?

>> If *any* free function `foo(shared T* bar)`, per your definition, is not threadsafe, then no other function with shared argument(s) can be threadsafe at all. So how do you call functions on shared data then? You keep saying "methods, methods..."
>>
>> struct Other { /* ... */ }
>>
>> struct S {
>>     void foo(shared Other*) shared;
>> }
>>
>> Per your rules, there would be *nothing* in the language to prevent calling S.foo with an unshared Other.
>
> That's true. And you can't do anything to it, so that's fine.

Yes you can do "anything" to it. If you couldn't, you wouldn't be able to implement `shared` at all. Forbidding reads and writes isn't enough to guarantee that you "can't do anything with it". *Unless* you forbid implicit conversion from mutable to shared. Then, and only then, your statement can hold.

>> So the only way to make your proposal work would be to forbid all functions from taking `shared T*` or `ref shared T` argument.
>
> No. Please read this thread again. From the beginning, every word.

Are you kidding me? Maybe it's *you* who should do that?..

> Actually, don't do that, because Manu's proposal is simple and elegant:

>> 1. the rule must be applied that shared object can not be read or written

No objection there, I fully support that. I even stated multiple times how it can be extended and why.

>> 2. attributing a method shared is a statement and a promise that the method is threadsafe

No objection here either.

>> The rest just follows naturally.

Nothing follows naturally. The proposal doesn't talk at all about the fact that you can't have "methods" on primitives, that you can't distinguish between shared and unshared data if that proposal is realized, that you absolutely destroy D's TLS-by-default treatment...

> There's actually one more thing: The one and only thing you can do (without unsafe casting) with a shared object, is call shared methods and free functions on it.

Functions that you must not be allowed to write per this same proposal. How quaint.

>> To sum up, things you implied but never specified in your proposal:
>>
>> 1. Primitive types can't be explicitly `shared`.
>
> Sure they can, they just can't present a thread-safe interface, so you can't do anything with a shared(int).

Ergo... you can't have functions taking pointers to shared primitives. Ergo, `shared <primitive type>` becomes a useless language construct.

>> 2. Free functions taking `shared` arguments are not allowed.
>
> Yes, they are. They would be using other shared methods or free functions on the shared argument, and would thus be thread-safe. If defined in the same module as the type on which they operate, they would have access to the internal state of the object, and would have to be written in such a way as to not violate the thread-safety of other methods and free functions that operate on it.

This contradicts (1). Either you can have functions taking shared T* arguments, thus
creating threadsafe interface for them, or you can't. If, per (1) as you say, you can't

>
>> 3. Only `shared` methods can implement threadsafe operations on `shared` data (which contradicts (2) already) <- this one you did specify.
>
> Non-shared methods are perfectly free to be thread-safe (and they should be, in the sense that they shouldn't interfere with shared methods). A better way to state this is that only shared methods may be called on a shared object. A shared object may also be passed to a function taking a shared parameter.
>
>
>> 4. Every variable is implicitly shared, whether intended so or not.

> Well, yes, in the same sense that every variable is also implicitly const, whether intended so or not.

I sort of expected that answer. No, nothing is implicitly const. When you pass a reference to a function taking const, *you keep mutable reference*, the function agrees to that, and it's only "promise" is to not modify data through the reference you gave it. But *you still keep mutable reference*. Just as you would keep *unshared mutable* reference if implicit conversion from mutable to shared existed.

On Thursday, 18 October 2018 at 12:15:07 UTC, Stanislav Blinov wrote:
> On Thursday, 18 October 2018 at 11:35:21 UTC, Simen Kjærås wrote:
>> On Thursday, 18 October 2018 at 10:08:48 UTC, Stanislav Blinov wrote:
>>> Manu,
>>>
>>> how is it that you can't see what *your own* proposal means??? Implicit casting from mutable to shared means that everything is shared by default! Precisely the opposite of what D proclaims.
>>
>> Well, sorta. But that's not a problem, because you can't do anything that's not threadsafe to something that's shared.
>
> Yes you can. You silently agree to another function's assumption that you pass shared data, while actually passing thread-local data and keeping treating it as thread-local. I.e. you silently agree to a race.

No, you don't. If I give you a locked box with no obvious way to open it, I can expect you not to open it. It's the same thing. If you have a shared(T), and it doesn't define a thread-safe interface, you can do nothing with it. If you are somehow able to cause a race with something with which you can do nothing, please tell me how, because I'm pretty sure that implies the very laws of logic are invalid.


>>> If *any* free function `foo(shared T* bar)`, per your definition, is not threadsafe, then no other function with shared argument(s) can be threadsafe at all. So how do you call functions on shared data then? You keep saying "methods, methods..."
>>>
>>> struct Other { /* ... */ }
>>>
>>> struct S {
>>>     void foo(shared Other*) shared;
>>> }
>>>
>>> Per your rules, there would be *nothing* in the language to prevent calling S.foo with an unshared Other.
>>
>> That's true. And you can't do anything to it, so that's fine.
>
> Yes you can do "anything" to it.

No, you can't. You can do thread-safe things to it. That's nothing, *unless* Other defines a shared (thread-safe) interface, in which case it's safe, and everything is fine.

Example:

struct Other {
    private Data payload;

    // shared function. Thread-safe, can be called from a
    // shared object, or from an unshared object.
    void twiddle() shared { payload.doSharedStuff(); }

    // unshared function. Cannot be called from a shared object.
    // Promises not to interfere with shared data, or to so only
    // in thread-safe ways (by calling thread-safe methods, or
    // by taking a mutex or equivalent).
    void twaddle() { payload.doSharedThings(); }

    // Bad function. Promises not to interfere with shared data,
    // but does so anyway.
    // Give the programmer a stern talking-to.
    void twank() {
        payload.fuckWith();
    }
}

struct S {
   void foo(shared Other* o) shared {
       // No can do - can't call non-shared functions on shared object.
       // o.twaddle();

       // Can do - twiddle is always safe to call.
       o.twiddle();
   }
}

> If you couldn't, you wouldn't be able to implement `shared` at all. Forbidding reads and writes isn't enough to guarantee that you "can't do anything with it".

Alright, so I have this shared object that I can't read from, and can't write to. It has no public shared members. What can I do with it? I can pass it to other guys, who also can't do anything with it. Are there other options?


>>> The rest just follows naturally.
>
> Nothing follows naturally. The proposal doesn't talk at all about the fact that you can't have "methods" on primitives,

You can't have thread-safe methods operating directly on primitives, because they already present a non-thread-safe interface. This is true. This follows naturally from the rules.


> that you can't distinguish between shared and unshared data if that proposal is realized,

And you can't do that currently either. Just like today, shared(T) means the T may or may not be shared with other thread. Nothing more, nothing less.


> that you absolutely destroy D's TLS-by-default treatment...

I'm unsure what you mean by this.


>> There's actually one more thing: The one and only thing you can do (without unsafe casting) with a shared object, is call shared methods and free functions on it.
>
> Functions that you must not be allowed to write per this same proposal. How quaint.

What? Which functions can't I write?

// Safe, regular function operating on shared data.
void foo(shared(Other)* o) {
    o.twiddle(); // Works great!
}

// Unsafe function. Should belong somewhere deep in druntime
// and only be used by certified wizards.
void bar(shared(int)* i) {
    atomicOp!"++"(i);
}

>>> 1. Primitive types can't be explicitly `shared`.
>>
>> Sure they can, they just can't present a thread-safe interface, so you can't do anything with a shared(int).
>
> Ergo... you can't have functions taking pointers to shared primitives. Ergo, `shared <primitive type>` becomes a useless language construct.

Yup, this is correct. But wrap it in a struct, like e.g. Atomic!int, and everything's hunky-dory.


>>> 2. Free functions taking `shared` arguments are not allowed.
>>
>> Yes, they are. They would be using other shared methods or free functions on the shared argument, and would thus be thread-safe. If defined in the same module as the type on which they operate, they would have access to the internal state of the object, and would have to be written in such a way as to not violate the thread-safety of other methods and free functions that operate on it.
>
> This contradicts (1). Either you can have functions taking shared T* arguments, thus
> creating threadsafe interface for them, or you can't. If, per (1) as you say, you can't

I have no idea where I or Manu have said you can't make functions that take shared(T)*.


>>> 4. Every variable is implicitly shared, whether intended so or not.
>
>> Well, yes, in the same sense that every variable is also implicitly const, whether intended so or not.
>
> I sort of expected that answer. No, nothing is implicitly const. When you pass a reference to a function taking const, *you keep mutable reference*, the function agrees to that, and it's only "promise" is to not modify data through the reference you gave it. But *you still keep mutable reference*. Just as you would keep *unshared mutable* reference if implicit conversion from mutable to shared existed.

Yup, and that's perfectly fine, because 'shared' means 'thread-safe'. I think Manu might have mentioned that once.

If a type presents both a shared and a non-shared interface, and the non-shared interface may do things that impact the shared part, these things must be done in a thread-safe manner. If that's not the case, you have a bug. The onus is on the creator of a type to do this.

Let's say it together: for a type to be thread-safe, all of its public members must be written in a thread-safe way. If a non-shared method may jeopardize this, the type is not thread-safe, and shouldn't provide a shared interface.

--
  Simen

On Thursday, 18 October 2018 at 13:09:10 UTC, Simen Kjærås wrote:
>> Ergo... you can't have functions taking pointers to shared primitives. Ergo, `shared <primitive type>` becomes a useless language construct.
>
> Yup, this is correct. But wrap it in a struct, like e.g. Atomic!int, and everything's hunky-dory.

Sorry, small mistake here. You're correct that shared <primitive type> becomes useless, except as a way to signal that there's something there, and you can't touch it. I was not replying to the part saying 'you can't have functions taking pointers to shared primitives'. That's just patently false. However, I don't see why you'd want such a function, since it can't do anything with what you pass it.

--
  Simen

On 10/17/18 10:26 PM, Manu wrote:
> On Wed, Oct 17, 2018 at 6:50 PM Steven Schveighoffer via Digitalmars-d
>>
>> The implicit cast means that you have to look at more than just your
>> method. You have to look at the entire module, and figure out all the
>> interactions, to see if the thread safe method actually is thread safe.
>> That's programming by convention, and fully trusting the programmer.
> 
> I don't understand... how can the outer context affect the
> threadsafety of a properly encapsulated thing?

[snip]

> You need to take it for an intellectual spin. Show me how it's corrupt
> rather than just presenting discomfort with the idea in theory.
> You're addicted to some concepts that you've carried around for a long
> time. There is no value in requiring casts, they're just a funky
> smell, and force the user to perform potentially unsafe manual
> conversions, or interactions that they don't understand.

For example (your example):

struct NotThreadsafe
{
  private int x;
  void local()
  {
    ++x; // <- invalidates the method below, you violate the other
function's `shared` promise
  }
  void notThreadsafe() shared
  {
    atomicIncrement(&x);
  }
}

First, note the comment. I can't look ONLY at the implementation of "notThreadSafe" (assuming the function name is less of a giveaway) in order to guarantee that it's actually thread safe. I have to look at the WHOLE MODULE. Anything could potentially do what local() does. I added private to x to at least give the appearance of thread safety.

But on top of that, if I can't implicitly cast mutable to shared, then this ACTUALLY IS thread safe, as long as all the casting in the module is sound (easy to search and verify), and hopefully all the casting is encapsulated in primitives like you have written. Because someone on the outside would have to cast a mutable item into a shared item, and this puts the responsibility on them to make sure it works.

I'm ALL FOR having shared be completely unusable as-is unless you cast (thanks for confirming what I suspected in your last post). It's the implicit casting which I think makes things way more difficult, and completely undercuts the utility of the compiler's mechanical checking.

And on top of that, I WANT that implementation. If I know something is not shared, why would I ever want to use atomics on it? I don't like needlessly throwing away performance. This is how I would write it:

struct ThreadSafe
{
   private int x;
   void increment()
   {
      ++x; // I know this is not shared, so no reason to use atomics
   }
   void increment() shared
   {
      atomicIncrement(&x); // use atomics, to avoid races
   }
}

The beauty of shared not being implicitly castable, is it allows you to focus on the implementation at hand, with the knowledge that nothing else can meddle with it. The goal of mechanical checking should be to narrow the focus of what needs to be proven correct.

-Steve

On 10/18/18 2:20 AM, Manu wrote:
> On Wed, Oct 17, 2018 at 5:05 AM Timon Gehr via Digitalmars-d
> <digitalmars-d@puremagic.com> wrote:
>>
>> [... all text ...]
> 
> OMFG, I just spent about 3 hours writing a super-detailed reply to all
> of Timon's posts in aggregate... I clicked send... and it's gone.
> I don't know if this is a gmail thing, a mailing list thing... no
> idea... but it's... gone.
> I can't repeat that effort :(
> 

If it's gmail, it should be in sent folder, no?

I've never had a gmail message that got sent fail to go into the sent box.

-Steve