On 10/17/18 10:33 AM, Nicholas Wilson wrote:
> On Wednesday, 17 October 2018 at 14:26:43 UTC, Timon Gehr wrote:
>> On 17.10.2018 16:14, Nicholas Wilson wrote:
>>>
>>> I was thinking that mutable -> shared const as apposed to mutable -> shared would get around the issues that Timon posted.
>>
>> Unfortunately not. For example, the thread with the mutable reference is not obliged to actually make the changes that are performed on that reference visible to other threads.
> 
> Yes, but that is covered by not being able to read non-atomically from a shared reference.

All sides must participate in synchronization for it to make sense. The mutable side has no obligation to use atomics. It can use ++data, and race conditions will happen.

-Steve

On Wednesday, 17 October 2018 at 15:51:04 UTC, Steven Schveighoffer wrote:
> On 10/17/18 9:58 AM, Nicholas Wilson wrote:
>> On Wednesday, 17 October 2018 at 13:25:28 UTC, Steven Schveighoffer wrote:
>>> It's identical to the top one. You now have a new unshared reference to shared data. This is done WITHOUT any agreed-upon synchronization.
>> 
>> It isn't, you typo'd it (I originally missed it too).
>>> int *p3 = cast(int*)p2;
>> 
>> vs
>> 
>>> int *p3 = p;
>
> It wasn't a typo.

The first example assigns p2, the second assigns p (which is thread local) _not_ p2 (which is shared), I'm confused.

On Wednesday, 17 October 2018 at 14:44:19 UTC, Guillaume Piolat wrote:

> The fact that this _type constructor_ finds its way into _identifiers_ create some concern: https://github.com/dlang/phobos/blob/656798f2b385437c239246b59e0433148190938c/std/experimental/allocator/package.d#L642

Well, ISharedAllocator is indeed overboard, but at least `allocateShared` would've been a very useful identifier indeed.

On Wednesday, 17 October 2018 at 14:14:56 UTC, Nicholas Wilson wrote:
> On Wednesday, 17 October 2018 at 07:24:13 UTC, Stanislav Blinov wrote:
>> On Wednesday, 17 October 2018 at 05:40:41 UTC, Walter Bright wrote:
>>
>>> When Andrei and I came up with the rules for:
>>>
>>>    mutable
>>>    const
>>>    shared
>>>    const shared
>>>    immutable
>>>
>>> and which can be implicitly converted to what, so far nobody has found a fault in those rules...
>>
>> Here's one: shared -> const shared shouldn't be allowed. Mutable aliasing in single-threaded code is bad enough as it is.
>
> Could you explain that a bit more? I don't understand it, mutable -> const is half the reason const exists.

Yes, but `shared` is not `const`. This 'might change' rule is poisonous for shared data: you essentially create extra work for the CPU for very little gain. There's absolutely no reason to share a read-only half-constant anyway. Either give immutable, or just flat out copy.

> I was thinking that mutable -> shared const as apposed to mutable -> shared would get around the issues that Timon posted.

It wouldn't, as Timon already explained. Writes to not-`shared` mutables might just not be propagated beyond the core. Wouldn't happen on amd64, of course, but still. It's not about atomicity of reads, it just depends on architecture. On some systems you have to explicitly synchronize memory.

On 10/17/18 12:27 PM, Nicholas Wilson wrote:
> On Wednesday, 17 October 2018 at 15:51:04 UTC, Steven Schveighoffer wrote:
>> On 10/17/18 9:58 AM, Nicholas Wilson wrote:
>>> On Wednesday, 17 October 2018 at 13:25:28 UTC, Steven Schveighoffer wrote:
>>>> It's identical to the top one. You now have a new unshared reference to shared data. This is done WITHOUT any agreed-upon synchronization.
>>>
>>> It isn't, you typo'd it (I originally missed it too).
>>>> int *p3 = cast(int*)p2;
>>>
>>> vs
>>>
>>>> int *p3 = p;
>>
>> It wasn't a typo.
> 
> The first example assigns p2, the second assigns p (which is thread local) _not_ p2 (which is shared), I'm confused.
> 

Here they are again:

int *p;
shared int *p2 = p;
int *p3 = cast(int*)p2;

int *p;
shared int *p2 = p;
int *p3 = p;


I'll put some asserts in that show they accomplish the same thing:

assert(p3 is p2);
assert(p3 is p);
assert(p2 is p);

What the example demonstrates is that while you are trying to disallow implicit casting of a shared pointer to an unshared pointer, you have inadvertently allowed it by leaving behind an unshared pointer that is the same thing.

While we do implicitly allow mutable to cast to const, it's because const is a weak guarantee. It's a guarantee that the data may not change via *this* reference, but could change via other references.

Shared doesn't have the same characteristics. In order for a datum to be safely shared, it must be accessed with synchronization or atomics by ALL parties. If you have one party that can simply change it without those, you will get races.

That's why shared/unshared is more akin to mutable/immutable than mutable/const.

It's true that only one thread will have thread-local access. It's not valid any more than having one mutable alias to immutable data.

-Steve

On Wed, Oct 17, 2018 at 10:30 AM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 10/17/18 12:27 PM, Nicholas Wilson wrote:
> > On Wednesday, 17 October 2018 at 15:51:04 UTC, Steven Schveighoffer wrote:
> >> On 10/17/18 9:58 AM, Nicholas Wilson wrote:
> >>> On Wednesday, 17 October 2018 at 13:25:28 UTC, Steven Schveighoffer wrote:
> >>>> It's identical to the top one. You now have a new unshared reference to shared data. This is done WITHOUT any agreed-upon synchronization.
> >>>
> >>> It isn't, you typo'd it (I originally missed it too).
> >>>> int *p3 = cast(int*)p2;
> >>>
> >>> vs
> >>>
> >>>> int *p3 = p;
> >>
> >> It wasn't a typo.
> >
> > The first example assigns p2, the second assigns p (which is thread
> > local) _not_ p2 (which is shared), I'm confused.
> >
>
> Here they are again:
>
> int *p;
> shared int *p2 = p;
> int *p3 = cast(int*)p2;
>
> int *p;
> shared int *p2 = p;
> int *p3 = p;
>
>
> I'll put some asserts in that show they accomplish the same thing:
>
> assert(p3 is p2);
> assert(p3 is p);
> assert(p2 is p);
>
> What the example demonstrates is that while you are trying to disallow implicit casting of a shared pointer to an unshared pointer, you have inadvertently allowed it by leaving behind an unshared pointer that is the same thing.

This doesn't make sense... you're showing a thread-local program.
The thread owning the unshared pointer is entitled to the unshared
pointer. It can make as many copies at it likes. They are all
thread-local.
There's only one owning thread, and you can't violate that without unsafe casts.

> In order for a datum to be
> safely shared, it must be accessed with synchronization or atomics by
> ALL parties.

** Absolutely **

> If you have one party that can simply change it without those, you will get races.

*** THIS IS NOT WHAT I'M PROPOSING ***

I've explained it a few times now, but people aren't reading what I
actually write, and just assume based on what shared already does that
they know what I'm suggesting.
You need to eject all presumptions from your mind, take the rules I
offer as verbatim, and do thought experiments from there.

> That's why shared/unshared is more akin to mutable/immutable than mutable/const.

Only if you misrepresent my suggestion.

> It's true that only one thread will have thread-local access. It's not valid any more than having one mutable alias to immutable data.

And this is why the immutable analogy is invalid. It's like const.
shared offers restricted access (like const), not a different class of
thing.
There is one thread with thread-local access, and many threads with
shared access.

If a shared (threadsafe) method can be defeated by threadlocal access, then it's **not threadsafe**, and the program is invalid.

struct NotThreadsafe
{
  int x;
  void local()
  {
    ++x; // <- invalidates the method below, you violate the other
function's `shared` promise
  }
  void notThreadsafe() shared
  {
    atomicIncrement(&x);
  }
}

struct Atomic(T)
{
  void opUnary(string op : "++")() shared { atomicIncrement(&val); }
  private T val;
}
struct Threadsafe
{
  Atomic!int x;
  void local()
  {
    ++x;
  }
  void threadsafe() shared
  {
    ++x;
  }
}

Naturally, local() is redundant, and it's perfectly fine for a
thread-local to call threadsafe() via implicit conversion.

Here's another one, where only a subset of the object is modeled to be threadsafe (this is particularly interesting to me):

struct Threadsafe
{
  int x;
  Atomic!int y;

  void notThreadsafe()
  {
    ++x;
    ++y;
  }
  void threadsafe() shared
  {
    ++y;
  }
}

In these examples, the thread-local function *does not* undermine the
threadsafety of threadsafe(), it MUST NOT undermine the threadsafety
of threadsafe(), or else threadsafe() **IS NOT THREADSAFE**.
In the second example, you can see how it's possible and useful to do
thread-local work without invalidating the objects threadsafety
commitments.


I've said this a bunch of times, there are 2 rules:
1. shared inhibits read and write access to members
2. `shared` methods must be threadsafe

>From there, shared becomes interesting and useful.

On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:

> I've said this a bunch of times, there are 2 rules:
> 1. shared inhibits read and write access to members
> 2. `shared` methods must be threadsafe
>
>>From there, shared becomes interesting and useful.

Oh God...

void atomicInc(shared int* i) { /* ... */ }

Now what? There are no "methods" for ints, only UFCS. Those functions can be as safe as you like, but if you allow implicit promotion of int* to shared int*, you *allow implicit races*.

On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:
>
> > I've said this a bunch of times, there are 2 rules:
> > 1. shared inhibits read and write access to members
> > 2. `shared` methods must be threadsafe
> >
> >>From there, shared becomes interesting and useful.
>
> Oh God...
>
> void atomicInc(shared int* i) { /* ... */ }
>
> Now what? There are no "methods" for ints, only UFCS. Those functions can be as safe as you like, but if you allow implicit promotion of int* to shared int*, you *allow implicit races*.

This function is effectively an intrinsic. It's unsafe by definition.
It's a tool for implementing threadsafe machinery.
No user can just start doing atomic operations on random ints and say
"it's threadsafe", you must encapsulate the threadsafe functionality
into some sort of object that aggregates all concerns and presents an
intellectually sound api.


Let me try one:

void free(void*) { ... }

Now what? I might have dangling pointers... it's a catastrophe!
It's essentially the same argument.
This isn't a function that professes to do something that people might
misunderstand and try to use in an unsafe way, it's a low-level
implementation device, which is used to build larger *useful*
constructs.

On 10/17/18 2:46 PM, Manu wrote:
> On Wed, Oct 17, 2018 at 10:30 AM Steven Schveighoffer via

>> What the example demonstrates is that while you are trying to disallow
>> implicit casting of a shared pointer to an unshared pointer, you have
>> inadvertently allowed it by leaving behind an unshared pointer that is
>> the same thing.
> 
> This doesn't make sense... you're showing a thread-local program.
> The thread owning the unshared pointer is entitled to the unshared
> pointer. It can make as many copies at it likes. They are all
> thread-local.

It's assumed that shared int pointer can be passed to another thread, right? Do I have to write a full program to demonstrate?

> There's only one owning thread, and you can't violate that without unsafe casts.

The what is the point of shared? Like why would you share data that NOBODY CAN USE?

At SOME POINT, shared data needs to be readable and writable. Any correct system is going to dictate how that works. It's a good start to make shared data unusable unless you cast. But then to make it implicitly castable from unshared defeats the whole purpose.

>> In order for a datum to be
>> safely shared, it must be accessed with synchronization or atomics by
>> ALL parties.
> 
> ** Absolutely **
> 
>> If you have one party that can simply change it without
>> those, you will get races.
> 
> *** THIS IS NOT WHAT I'M PROPOSING ***
> 
> I've explained it a few times now, but people aren't reading what I
> actually write, and just assume based on what shared already does that
> they know what I'm suggesting.
> You need to eject all presumptions from your mind, take the rules I
> offer as verbatim, and do thought experiments from there.

What seems to be a mystery here is how one is to actually manipulate shared data. If it's not usable as shared data, how does one use it?

> 
>> That's why shared/unshared is more akin to mutable/immutable than
>> mutable/const.
> 
> Only if you misrepresent my suggestion.

It's not misrepresentation, I'm trying to fill in the holes with the only logical possibilities I can think of.

> 
>> It's true that only one thread will have thread-local access. It's not
>> valid any more than having one mutable alias to immutable data.
> 
> And this is why the immutable analogy is invalid. It's like const.
> shared offers restricted access (like const), not a different class of
> thing.

No, not at all. Somehow one must manipulate shared data. If shared data cannot be read or written, there is no reason to share it.

So LOGICALLY, we have to assume, yes there actually IS a way to manipulate shared data through these very carefully constructed and guarded things.

> There is one thread with thread-local access, and many threads with
> shared access.
> 
> If a shared (threadsafe) method can be defeated by threadlocal access,
> then it's **not threadsafe**, and the program is invalid.
> 
> struct NotThreadsafe
> {
>    int x;
>    void local()
>    {
>      ++x; // <- invalidates the method below, you violate the other
> function's `shared` promise
>    }
>    void notThreadsafe() shared
>    {
>      atomicIncrement(&x);
>    }
> }

So the above program is invalid. Is it compilable with your added allowance of implicit casting to shared? If it's not compilable, why not? If it is compilable, how in the hell does your proposal help anything? I get the exact behavior today without any changes (except today, I need to explicitly cast, which puts the onus on me).

> 
> struct Atomic(T)
> {
>    void opUnary(string op : "++")() shared { atomicIncrement(&val); }
>    private T val;
> }
> struct Threadsafe
> {
>    Atomic!int x;
>    void local()
>    {
>      ++x;
>    }
>    void threadsafe() shared
>    {
>      ++x;
>    }
> }
> 
> Naturally, local() is redundant, and it's perfectly fine for a
> thread-local to call threadsafe() via implicit conversion.

In this case, yes. But that's not because of anything the compiler can prove.

How does Atomic work? I thought shared data was not usable? I'm being pedantic because every time I say "well at some point you must be able to modify things", you explode.

Complete the sentence: "In order to read or write shared data, you have to ..."

> 
> Here's another one, where only a subset of the object is modeled to be
> threadsafe (this is particularly interesting to me):
> 
> struct Threadsafe
> {
>    int x;
>    Atomic!int y;
> 
>    void notThreadsafe()
>    {
>      ++x;
>      ++y;
>    }
>    void threadsafe() shared
>    {
>      ++y;
>    }
> }
> 
> In these examples, the thread-local function *does not* undermine the
> threadsafety of threadsafe(), it MUST NOT undermine the threadsafety
> of threadsafe(), or else threadsafe() **IS NOT THREADSAFE**.
> In the second example, you can see how it's possible and useful to do
> thread-local work without invalidating the objects threadsafety
> commitments.
> 
> 
> I've said this a bunch of times, there are 2 rules:
> 1. shared inhibits read and write access to members
> 2. `shared` methods must be threadsafe
> 
>>From there, shared becomes interesting and useful.
> 

Given rule 1, how does Atomic!int actually work, if it can't read or write shared members?
For rule 2, how does the compiler actually prove this?

Any programming by convention, we can do today. We can implement Atomic!int with the current compiler, using unsafe casts inside @trusted blocks.

-Steve

I don't see any problem with this proposal as long as these points hold:

- Shared <-> Unshared is never implicit, either requiring an explicit cast (both ways) or having a language support which allows the conversion gracefully.
- Shared methods are called by compiler if the type is shared or if there is no unshared equivalent.
- Programmer needs to guarantee that shared -> unshared cast/conversion is thread-safe by hand; such as acquiring a lock, atomic operations...
- Programmer needs to guarantee that when unshared -> shared cast/conversion happens, data is not accessed through unshared reference during the lifetime of shared reference(s). Effectively this means a data needs to be treated as shared everywhere at the same time otherwise all things fall apart.