On 17.10.18 20:46, Manu wrote:
> struct NotThreadsafe
> {
>    int x;
>    void local()
>    {
>      ++x; // <- invalidates the method below, you violate the other
> function's `shared` promise
>    }
>    void notThreadsafe() shared
>    {
>      atomicIncrement(&x);
>    }
> }

In the `shared` method you'd get a nice error when attempting `++x;`, because it's not thread-safe. With your proposal, it's just as unsafe in `local`, but you don't get any help from the compiler in spotting it. `local` is effectively `@trusted` without being marked as such.

On Wednesday, 17 October 2018 at 19:25:33 UTC, Manu wrote:
> On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>
>> On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:
>>
>> > I've said this a bunch of times, there are 2 rules:
>> > 1. shared inhibits read and write access to members
>> > 2. `shared` methods must be threadsafe
>> >
>> >>From there, shared becomes interesting and useful.
>>
>> Oh God...
>>
>> void atomicInc(shared int* i) { /* ... */ }
>>
>> Now what? There are no "methods" for ints, only UFCS. Those functions can be as safe as you like, but if you allow implicit promotion of int* to shared int*, you *allow implicit races*.
>
> This function is effectively an intrinsic. It's unsafe by definition.

Only if implicit conversion is allowed. If it isn't, that's likely @trusted, and this:

void atomicInc(ref shared int);

can even be @safe.

> It's a tool for implementing threadsafe machinery.
> No user can just start doing atomic operations on random ints and say
> "it's threadsafe", you must encapsulate the threadsafe functionality
> into some sort of object that aggregates all concerns and presents an
> intellectually sound api.

Threadsafety starts and ends with the programmer. By your logic *all* functions operating on `shared` are unsafe then. As far as compiler is concerned, there would be no difference between these two:

struct S {}
void atomicInc(ref shared S);

and

struct S { void atomicInc() shared { /* ... */ } }

The signatures of those two functions are exactly the same. How is that different from a function taking a shared int pointer or reference?

>
> Let me try one:
>
> void free(void*) { ... }
>
> Now what? I might have dangling pointers... it's a catastrophe!

One could argue that it should be void free(ref void* p) { /* ... */ p = null; }
As a matter of fact, in my own allocators memory blocks allocated by them are passed by value and are non-copyable, they're not just void[] as in std.experimental.allocator. One must 'move' them to pass ownership, and that includes deallocation. But that's another story altogether.

> It's essentially the same argument.
> This isn't a function that professes to do something that people might
> misunderstand and try to use in an unsafe way, it's a low-level
> implementation device, which is used to build larger *useful*
> constructs.

You're missing the point, again. You have an int. You pass a pointer to it to some API that takes an int*. You continue to use your int as just an int. The API changes, and now the function you called previously takes a shared int*. Implicit conversion works, everything compiles, you have a race. Now, that's of course an extremely stupid scenario. The point is: the caller of some API *must* assert that they indeed pass shared data. It's insufficient for the API alone to "promise" taking shared data. That's the difference with promotion to `const`.

On Wed, Oct 17, 2018 at 12:35 PM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 10/17/18 2:46 PM, Manu wrote:
> > On Wed, Oct 17, 2018 at 10:30 AM Steven Schveighoffer via
>
> >> What the example demonstrates is that while you are trying to disallow implicit casting of a shared pointer to an unshared pointer, you have inadvertently allowed it by leaving behind an unshared pointer that is the same thing.
> >
> > This doesn't make sense... you're showing a thread-local program. The thread owning the unshared pointer is entitled to the unshared pointer. It can make as many copies at it likes. They are all thread-local.
>
> It's assumed that shared int pointer can be passed to another thread, right? Do I have to write a full program to demonstrate?

And that shared(int)* provides no access. No other thread with that
pointer can do anything with it.

> > There's only one owning thread, and you can't violate that without unsafe casts.
>
> The what is the point of shared? Like why would you share data that NOBODY CAN USE?

You can call shared methods. They promise threadsafety.
That's a small subset of the program, but that's natural; only a very
small subset of the program is safe to be called from a shared
context.

In addition, traditional unsafe interactions which may involve
acquiring locks and doing casts remain exactly the same, and the exact
same design patterns must apply which assure that the object is
handled correctly.
I'm not suggesting any changes that affect that workflow.

> At SOME POINT, shared data needs to be readable and writable. Any correct system is going to dictate how that works. It's a good start to make shared data unusable unless you cast. But then to make it implicitly castable from unshared defeats the whole purpose.

No. No casting! This is antiquated workflow.. I'm not trying to take
it away from you, but it's not an interesting model for the future.
`shared` can model more than just that.
You can call threadsafe methods. Shared methods explicitly dictate how
the system works, and in a very clear and obvious/intuitive way.

The implicit cast makes using threadsafe objects more convenient when you only have one, which is extremely common.

> >> In order for a datum to be
> >> safely shared, it must be accessed with synchronization or atomics by
> >> ALL parties.
> >
> > ** Absolutely **
> >
> >> If you have one party that can simply change it without those, you will get races.
> >
> > *** THIS IS NOT WHAT I'M PROPOSING ***
> >
> > I've explained it a few times now, but people aren't reading what I
> > actually write, and just assume based on what shared already does that
> > they know what I'm suggesting.
> > You need to eject all presumptions from your mind, take the rules I
> > offer as verbatim, and do thought experiments from there.
>
> What seems to be a mystery here is how one is to actually manipulate shared data. If it's not usable as shared data, how does one use it?

Call shared methods. It's not like I haven't been saying this in every
post since my OP.
The only possible thing that you can safely do with a shared object is
call a method that has been carefully designed for thread-safe
calling. Any other access is invalid under any circumstance.

> >> It's true that only one thread will have thread-local access. It's not valid any more than having one mutable alias to immutable data.
> >
> > And this is why the immutable analogy is invalid. It's like const. shared offers restricted access (like const), not a different class of thing.
>
> No, not at all. Somehow one must manipulate shared data. If shared data cannot be read or written, there is no reason to share it.

Call shared methods.

> So LOGICALLY, we have to assume, yes there actually IS a way to manipulate shared data through these very carefully constructed and guarded things.

Yes, call the methods, they were carefully constructed to be threadsafe. Only functions that have made a promise to be threadsafe and implemented that complexity are valid interactions.

> > There is one thread with thread-local access, and many threads with shared access.
> >
> > If a shared (threadsafe) method can be defeated by threadlocal access, then it's **not threadsafe**, and the program is invalid.
> >
> > struct NotThreadsafe
> > {
> >    int x;
> >    void local()
> >    {
> >      ++x; // <- invalidates the method below, you violate the other
> > function's `shared` promise
> >    }
> >    void notThreadsafe() shared
> >    {
> >      atomicIncrement(&x);
> >    }
> > }
>
> So the above program is invalid. Is it compilable with your added allowance of implicit casting to shared? If it's not compilable, why not?

All my examples assume my implicit conversion rule. But regardless, under my proposal, the above program is invalid. This violates the threadsafe promise.

> If it is compilable, how in the hell does your proposal help anything? I get the exact behavior today without any changes (except today, I need to explicitly cast, which puts the onus on me).

My proposal doesn't help this program, it's invalid. I'm just demonstrating what an invalid program looks like.

> > struct Atomic(T)
> > {
> >    void opUnary(string op : "++")() shared { atomicIncrement(&val); }
> >    private T val;
> > }
> > struct Threadsafe
> > {
> >    Atomic!int x;
> >    void local()
> >    {
> >      ++x;
> >    }
> >    void threadsafe() shared
> >    {
> >      ++x;
> >    }
> > }
> >
> > Naturally, local() is redundant, and it's perfectly fine for a
> > thread-local to call threadsafe() via implicit conversion.
>
> In this case, yes. But that's not because of anything the compiler can prove.

The compiler can't 'prove' anything at all related to threadsafety. We
need to make one assumption; that `shared` methods are expected to be
threadsafe, and from there the compiler can prove correctness with
respect to that assumption.
In this case, `Atomic.opUnary("++")` promises that it's threadsafe,
and as such, the aggregate can safely use that tool to implement
higher-level logic.

> How does Atomic work? I thought shared data was not usable? I'm being pedantic because every time I say "well at some point you must be able to modify things", you explode.

Atomic implements a safe utility using unsafe primitives (atomic
increment intrinsic).
Atomic wraps the unsafe call to an intrinsic into a box that's safe,
and can be used by clients.
In my worldview, atomic is at the bottom of the chain-of-trust. It's
effectively a @trusted implementation of a foundational tool.
Almost every low level tool is of this nature.

> Complete the sentence: "In order to read or write shared data, you have to ..."

Call a shared method, or at the bottom of the stack, you need to do unsafe (@trusted?) implementations of the foundational machinery (possibly using casts), and package into boxes that are safe to interact with and build out from.

> > Here's another one, where only a subset of the object is modeled to be threadsafe (this is particularly interesting to me):
> >
> > struct Threadsafe
> > {
> >    int x;
> >    Atomic!int y;
> >
> >    void notThreadsafe()
> >    {
> >      ++x;
> >      ++y;
> >    }
> >    void threadsafe() shared
> >    {
> >      ++y;
> >    }
> > }
> >
> > In these examples, the thread-local function *does not* undermine the
> > threadsafety of threadsafe(), it MUST NOT undermine the threadsafety
> > of threadsafe(), or else threadsafe() **IS NOT THREADSAFE**.
> > In the second example, you can see how it's possible and useful to do
> > thread-local work without invalidating the objects threadsafety
> > commitments.
> >
> >
> > I've said this a bunch of times, there are 2 rules:
> > 1. shared inhibits read and write access to members
> > 2. `shared` methods must be threadsafe
> >
> >>From there, shared becomes interesting and useful.
> >
>
> Given rule 1, how does Atomic!int actually work, if it can't read or write shared members?

It's an intrinsic. You use it the same as malloc() or free(). It's a
piece of low-level mechanical tooling which you use in an unsafe way,
but you then wrap it in a layer that introduces the type-safety.
malloc() returns a void*, which you cast to the intended type, and
then perform construction.
You can't implement a typesafe new without malloc() at the bottom of the stack.

You need to raise your vision one-level higher to users of Atomic to see interesting interactions.

> For rule 2, how does the compiler actually prove this?
>
> Any programming by convention, we can do today. We can implement Atomic!int with the current compiler, using unsafe casts inside @trusted blocks.

It can't. I don't know what can be done to mechanically enforce this
requirement, but I would suggest that it's a goal to work towards in
the future with any technology possible.
In the meantime though, if we accept that the user writing a
threadsafe tool is responsible for delivering on their promise, then
the system that emerges is widely useful. The higher-level becomes
generally interesting, the low-level will remain to be implemented by
experts, and is no change from the situation today.
The low level doesn't really care much about type-safety, it's the
high-level I'm interested in. We can tell a MUCH better story about
how users can interact with shared machinery, and we should.

On Wed, Oct 17, 2018 at 2:15 PM Stanislav Blinov via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On Wednesday, 17 October 2018 at 19:25:33 UTC, Manu wrote:
> > On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>
> >> On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:
> >>
> >> > I've said this a bunch of times, there are 2 rules:
> >> > 1. shared inhibits read and write access to members
> >> > 2. `shared` methods must be threadsafe
> >> >
> >> >>From there, shared becomes interesting and useful.
> >>
> >> Oh God...
> >>
> >> void atomicInc(shared int* i) { /* ... */ }
> >>
> >> Now what? There are no "methods" for ints, only UFCS. Those functions can be as safe as you like, but if you allow implicit promotion of int* to shared int*, you *allow implicit races*.
> >
> > This function is effectively an intrinsic. It's unsafe by definition.
>
> Only if implicit conversion is allowed. If it isn't, that's likely @trusted, and this:
>
> void atomicInc(ref shared int);
>
> can even be @safe.

In this case, with respect to the context (a single int) atomicInc()
is ALWAYS safe, even with implicit conversion. You can atomicInc() a
thread-local int perfectly safely.

> > It's a tool for implementing threadsafe machinery.
> > No user can just start doing atomic operations on random ints
> > and say
> > "it's threadsafe", you must encapsulate the threadsafe
> > functionality
> > into some sort of object that aggregates all concerns and
> > presents an
> > intellectually sound api.
>
> Threadsafety starts and ends with the programmer. By your logic *all* functions operating on `shared` are unsafe then. As far as compiler is concerned, there would be no difference between these two:
>
> struct S {}
> void atomicInc(ref shared S);
>
> and
>
> struct S { void atomicInc() shared { /* ... */ } }
>
> The signatures of those two functions are exactly the same. How is that different from a function taking a shared int pointer or reference?

It's not, atomicInc() of an int is always safe with respect to the int itself.
You can call atomicInc() on an unshared int and it's perfectly fine,
but now you need to consider context, and that's a problem for the
design of the higher-level scope.

To maintain thread-safety, the int in question must be appropriately contained.

The problem is that the same as the example I presented before, which I'll repeat:

struct InvalidProgram
{
  int x;
  void fun() { ++x; }
  void gun() shared { atomicInc(&x); }
}

The method gun() (and therefore the whole object) is NOT threadsafe by
my definition, because fun() violates the threadsafety of gun().
The situation applies equally here that:
int x;
atomicInc(&x);
++x; // <- by my definition, this 'API' (increment an int) violates
the threadsafety of atomicInc(), and atomicInc() is therefore not
threadsafe.

`int` doesn't present a threadsafe API, so int is by definition, NOT threadsafe. atomicInc() should be @system, and not @trusted.

If you intend to share an int, use Atomic!int, because it has a threadsafe API.
atomicInc(shared int*) is effectively just an unsafe intrinsic, and
its only use is at ground-level implementation of threadsafe
machinery, like malloc() and free().

> > Let me try one:
> >
> > void free(void*) { ... }
> >
> > Now what? I might have dangling pointers... it's a catastrophe!
>
> One could argue that it should be void free(ref void* p) { /* ...
> */ p = null; }

void *p2 = p;
free(p);
p2.crash();

> As a matter of fact, in my own allocators memory blocks allocated by them are passed by value and are non-copyable, they're not just void[] as in std.experimental.allocator. One must 'move' them to pass ownership, and that includes deallocation. But that's another story altogether.

Right, now you're talking about move semantics to implement transfer of ownership... you might recall I was arguing this exact case to express transferring ownership of objects between threads earlier. This talk of blunt casts and "making sure everything is good" is all just great, but it doesn't mean anything interesting with respect to `shared`. It should be interesting even without unsafe casts.

> > It's essentially the same argument.
> > This isn't a function that professes to do something that
> > people might
> > misunderstand and try to use in an unsafe way, it's a low-level
> > implementation device, which is used to build larger *useful*
> > constructs.
>
> You're missing the point, again. You have an int. You pass a pointer to it to some API that takes an int*. You continue to use your int as just an int.

You have written an invalid program. I can think of an infinite number
of ways to write an invalid program.
In this case, don't have an `int`, instead, have an Atomic!int; you
now guarantee appropriate access, problem solved!
If you do have an int, don't pass it to other threads at random when
you don't have any idea what they intend to do with it! That's basic
common sense. You don't pass a pointer to a function if you don't know
what it does with the pointer!

> The API changes, and now the function
> you called previously takes a shared int*. Implicit conversion
> works, everything compiles, you have a race. Now, that's of
> course an extremely stupid scenario.

Yes.

> The point is: the caller of
> some API *must* assert that they indeed pass shared data. It's
> insufficient for the API alone to "promise" taking shared data.
> That's the difference with promotion to `const`.

The caller doesn't care if it's true that the callee can't do anything
with it that's unsafe anyway.
We effect that state by removing all non-threadsafe access.

On Wednesday, 17 October 2018 at 23:12:48 UTC, Manu wrote:
> On Wed, Oct 17, 2018 at 2:15 PM Stanislav Blinov via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>
>> On Wednesday, 17 October 2018 at 19:25:33 UTC, Manu wrote:
>> > On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>> >>
>> >> On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:
>> >>
>> >> > I've said this a bunch of times, there are 2 rules:
>> >> > 1. shared inhibits read and write access to members
>> >> > 2. `shared` methods must be threadsafe
>> >> >
>> >> >>From there, shared becomes interesting and useful.
>> >>
>> >> Oh God...
>> >>
>> >> void atomicInc(shared int* i) { /* ... */ }
>> >>
>> >> Now what? There are no "methods" for ints, only UFCS. Those functions can be as safe as you like, but if you allow implicit promotion of int* to shared int*, you *allow implicit races*.
>> >
>> > This function is effectively an intrinsic. It's unsafe by definition.
>>
>> Only if implicit conversion is allowed. If it isn't, that's likely @trusted, and this:
>>
>> void atomicInc(ref shared int);
>>
>> can even be @safe.
>
> In this case, with respect to the context (a single int) atomicInc()
> is ALWAYS safe, even with implicit conversion. You can atomicInc() a
> thread-local int perfectly safely.

Yes, *you* can. *Another* function can't unless *you* allow for it to be safe. You can't do that if that function silently assumes you gave it shared data, when in fact you did not.

>> The signatures of those two functions are exactly the same. How is that different from a function taking a shared int pointer or reference?
>
> It's not, atomicInc() of an int is always safe with respect to the int itself.
> You can call atomicInc() on an unshared int and it's perfectly fine,
> but now you need to consider context, and that's a problem for  the
> design of the higher-level scope.
>
> To maintain thread-safety, the int in question must be appropriately contained.

Exactly. And that means it can't convert to shared without my say so :)

> The problem is that the same as the example I presented before, which I'll repeat:
>
> struct InvalidProgram
> {
>   int x;
>   void fun() { ++x; }
>   void gun() shared { atomicInc(&x); }
> }
>
> The method gun() (and therefore the whole object) is NOT threadsafe by
> my definition, because fun() violates the threadsafety of gun().
> The situation applies equally here that:
> int x;
> atomicInc(&x);
> ++x; // <- by my definition, this 'API' (increment an int) violates the threadsafety of atomicInc(), and atomicInc() is therefore not threadsafe.

No. The 'API' is just the atomicInc function. You, the user of that API, own the int. If the API wants a shared int from you, you have to be in agreement. You can't have any agreement if the API is only making promises and assumptions.

> `int` doesn't present a threadsafe API, so int is by definition, NOT threadsafe. atomicInc() should be @system, and not @trusted.

Exactly. `int` isn't threadsafe and therefore cannot automatically convert to `shared int`.

> If you intend to share an int, use Atomic!int, because it has a threadsafe API.

No. With current implementation of `shared`, which disallows your automatic promotion,
your intent is enforced. You cannot share a local `int` unless *you know* it's safe to do so and therefore can cast that int to shared.

> atomicInc(shared int*) is effectively just an unsafe intrinsic, and

It is only unsafe if you allow int* to silently convert to shared int*. If you can't do that, you can't call `atomicInc` on an int*.

>> One could argue that it should be void free(ref void* p) { /* ... */ p = null; }

> void *p2 = p;
> free(p);
> p2.crash();

That's exactly analogous to what you're proposing: leaking `shared` references while keeping unshared data.

>> As a matter of fact, in my own allocators memory blocks allocated by them are passed by value and are non-copyable, they're not just void[] as in std.experimental.allocator. One must 'move' them to pass ownership, and that includes deallocation. But that's another story altogether.
>
> Right, now you're talking about move semantics to implement transfer of ownership... you might recall I was arguing this exact case to express transferring ownership of objects between threads earlier.

You went off on an irrelevant tangent there, and I feel like you didn't even see my reply. You don't pass any ownership when you share. You just share. As an owner, you get to access the un-`shared` interface freely. Others do not.

> This talk of blunt casts and "making sure everything is good" is all just great, but it doesn't mean  anything interesting with respect to `shared`. It should be interesting even without unsafe casts.

Again, I feel like you didn't see my reply. It's not talk about just blunt casts to make sure everything is good. You either have shared data to begin with, and so can share it freely, or you *know* that you can share this particular piece of data, even if it itself isn't marked `shared`, and you *assert* that by deliberately casting. *You* know, *you* cast, not "some function expects you to know, and just auto-casts".

>> You're missing the point, again. You have an int. You pass a pointer to it to some API that takes an int*. You continue to use your int as just an int.

> You have written an invalid program. I can think of an infinite number of ways to write an invalid program.

No, I have not. I didn't make any promises that my data was shared, and I wasn't expecting it to be treated as such. I didn't even author that API. The other guy (the API) made wrong assumptions. Don't you see the difference here?

> In this case, don't have an `int`, instead, have an Atomic!int; you now guarantee appropriate access, problem solved!

No, it isn't. My int *is* thread-local, *I* don't need an Atomic!int, I didn't sanction that int's use as shared in any way. Yet the automatic conversion presumes that I have.

As I stated previously, there's no difference between Atomic!int and free functions operating on shared int* (or ref shared int). Struct methods are sugared versions of those free functions, *nothing more*. That's why we have UFCS.

> If you do have an int, don't pass it to other threads at random when you don't have any idea what they intend to do with it!

*I* don't pass it to other threads at random, and I expect other code to not do so without *my* approval. What approval can I give if *other* code can silently assume I'm giving it shared data, when in fact I'm not?

> That's basic common sense. You don't pass a pointer to a function if you don't know what it does with the pointer!

I should know what the function does with the pointer from it's signature. Now, currently in D that very blurry. *Hopefully* with 'scope', DIP25 and DIP1000 this becomes more common. But that's at least what we should strive for.
If a function takes `shared`, I better be sure I'm giving it `shared`. The only way to do so is either have `shared` to begin with, or explicitly cast when I know I can do so.

>> The point is: the caller of
>> some API *must* assert that they indeed pass shared data. It's
>> insufficient for the API alone to "promise" taking shared data.
>> That's the difference with promotion to `const`.

> The caller doesn't care if it's true that the callee can't do anything with it that's unsafe anyway.
> We effect that state by removing all non-threadsafe access.

You allow non-threadsafe access with implicit cast, not remove it.

This broken record is getting very tiresome... Let me ask you this once again: *why* are you so bent on this implicit conversion from mutable to shared? So far the only reason I've seen is to just avoid writing additional methods that forward to `shared` methods.

Most of your casts will be the other way around, will have to be explicit and there's nothing that can be done about that. You'll only have mutable->shared casts in few select cases, exactly because they're corner cases where you *need* to make the decision clear.

I don't have anything to add that hasn't been said yet but it's good to see some thinking on this subject. It feels like progress.

On 10/17/18 6:37 PM, Manu wrote:
> On Wed, Oct 17, 2018 at 12:35 PM Steven Schveighoffer via
> Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>
>> On 10/17/18 2:46 PM, Manu wrote:
>>> On Wed, Oct 17, 2018 at 10:30 AM Steven Schveighoffer via
>>
>>>> What the example demonstrates is that while you are trying to disallow
>>>> implicit casting of a shared pointer to an unshared pointer, you have
>>>> inadvertently allowed it by leaving behind an unshared pointer that is
>>>> the same thing.
>>>
>>> This doesn't make sense... you're showing a thread-local program.
>>> The thread owning the unshared pointer is entitled to the unshared
>>> pointer. It can make as many copies at it likes. They are all
>>> thread-local.
>>
>> It's assumed that shared int pointer can be passed to another thread,
>> right? Do I have to write a full program to demonstrate?
> 
> And that shared(int)* provides no access. No other thread with that
> pointer can do anything with it.

So then it's a misnomer -- it's not really shared, because I can't do anything with it.

> 
>>> There's only one owning thread, and you can't violate that without unsafe casts.
>>
>> The what is the point of shared? Like why would you share data that
>> NOBODY CAN USE?
> 
> You can call shared methods. They promise threadsafety.
> That's a small subset of the program, but that's natural; only a very
> small subset of the program is safe to be called from a shared
> context.

All I can see is that a shared method promises to be callable on shared or unshared data. In essence, it promises nothing.

It's the programmer who must implement the thread safety, and there really is no help at all from the compiler for this. At some level, there will be either casts, or intrinsics, both of which are unsafe without knowing all the context of the object. In any case, it's simply a false guarantee of thread safety, which might as well be a convention of "any function which starts with TS_ is supposed to be thread safe".

shared in the current form promises one thing and one thing only -- data marked as shared is actually sharable between threads, and data not marked as shared is actually not shared between threads. This new regime you are proposing does nothing extra or new, except break that guarantee.

>> At SOME POINT, shared data needs to be readable and writable. Any
>> correct system is going to dictate how that works. It's a good start to
>> make shared data unusable unless you cast. But then to make it
>> implicitly castable from unshared defeats the whole purpose.
> 
> No. No casting! This is antiquated workflow.. I'm not trying to take
> it away from you, but it's not an interesting model for the future.
> `shared` can model more than just that.
> You can call threadsafe methods. Shared methods explicitly dictate how
> the system works, and in a very clear and obvious/intuitive way.
> 
> The implicit cast makes using threadsafe objects more convenient when
> you only have one, which is extremely common.

The implicit cast means that you have to look at more than just your method. You have to look at the entire module, and figure out all the interactions, to see if the thread safe method actually is thread safe. That's programming by convention, and fully trusting the programmer.

I don't think this thread is going anywhere, so I'll just have to wait and see if someone else can explain it better. I'm a firm no on implicit casting from mutable to shared.

-Steve

On Wed, Oct 17, 2018 at 5:35 PM Stanislav Blinov via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On Wednesday, 17 October 2018 at 23:12:48 UTC, Manu wrote:
> > On Wed, Oct 17, 2018 at 2:15 PM Stanislav Blinov via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>
> >> On Wednesday, 17 October 2018 at 19:25:33 UTC, Manu wrote:
> >> > On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >> >>
> >> >> On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:
> >> >>
> >> >> > I've said this a bunch of times, there are 2 rules:
> >> >> > 1. shared inhibits read and write access to members
> >> >> > 2. `shared` methods must be threadsafe
> >> >> >
> >> >> >>From there, shared becomes interesting and useful.
> >> >>
> >> >> Oh God...
> >> >>
> >> >> void atomicInc(shared int* i) { /* ... */ }
> >> >>
> >> >> Now what? There are no "methods" for ints, only UFCS. Those functions can be as safe as you like, but if you allow implicit promotion of int* to shared int*, you *allow implicit races*.
> >> >
> >> > This function is effectively an intrinsic. It's unsafe by definition.
> >>
> >> Only if implicit conversion is allowed. If it isn't, that's likely @trusted, and this:
> >>
> >> void atomicInc(ref shared int);
> >>
> >> can even be @safe.
> >
> > In this case, with respect to the context (a single int)
> > atomicInc()
> > is ALWAYS safe, even with implicit conversion. You can
> > atomicInc() a
> > thread-local int perfectly safely.
>
> Yes, *you* can. *Another* function can't unless *you* allow for it to be safe. You can't do that if that function silently assumes you gave it shared data, when in fact you did not.
>
> >> The signatures of those two functions are exactly the same. How is that different from a function taking a shared int pointer or reference?
> >
> > It's not, atomicInc() of an int is always safe with respect to
> > the int itself.
> > You can call atomicInc() on an unshared int and it's perfectly
> > fine,
> > but now you need to consider context, and that's a problem for
> > the
> > design of the higher-level scope.
> >
> > To maintain thread-safety, the int in question must be appropriately contained.
>
> Exactly. And that means it can't convert to shared without my say so :)

It can though, because `int` doesn't have any shared methods; it's inaccessible. You can't do anything with a shared int, so it's safe to distribute regardless.

> > The problem is that the same as the example I presented before, which I'll repeat:
> >
> > struct InvalidProgram
> > {
> >   int x;
> >   void fun() { ++x; }
> >   void gun() shared { atomicInc(&x); }
> > }
> >
> > The method gun() (and therefore the whole object) is NOT
> > threadsafe by
> > my definition, because fun() violates the threadsafety of gun().
> > The situation applies equally here that:
> > int x;
> > atomicInc(&x);
> > ++x; // <- by my definition, this 'API' (increment an int)
> > violates the threadsafety of atomicInc(), and atomicInc() is
> > therefore not threadsafe.
>
> No. The 'API' is just the atomicInc function. You, the user of that API, own the int. If the API wants a shared int from you, you have to be in agreement. You can't have any agreement if the API is only making promises and assumptions.

Okay, here's an idea... atomicInc() should NOT take `shared int*`,
change atomicInt() to receive int*.
We agree that `int` is not a threadsafe type, atomicInc() is not a
threadsafe method, and therefore shouldn't be shared.

If you have a shared(int)*, and you want to do un-threadsafe
atomicInc(), you need to cast to int*.
I think this is actually correct by my definition, because we
recognise int is not a threadsafe type, and atomicInc() is not a
threadsafe method, so hard-cast is required, and programmer must
appropriately validate the context when doing an unsafe operation.

> > `int` doesn't present a threadsafe API, so int is by definition, NOT threadsafe. atomicInc() should be @system, and not @trusted.
>
> Exactly. `int` isn't threadsafe and therefore cannot automatically convert to `shared int`.

It can convert to shared int, because it's inaccessible... you can't do anything with a `shared int` except unsafe cast shared away. Once you're in unsafe land, you are responsible for assuring correct conditions.

> > If you intend to share an int, use Atomic!int, because it has a threadsafe API.
>
> No. With current implementation of `shared`, which disallows your
> automatic promotion,
> your intent is enforced. You cannot share a local `int` unless
> *you know* it's safe to do so and therefore can cast that int to
> shared.

Yes, but we're not talking about current definition of shared. Stop
confusing the conversation.
This thread is about the rules I define.

> > atomicInc(shared int*) is effectively just an unsafe intrinsic,
> > and
>
> It is only unsafe if you allow int* to silently convert to shared int*. If you can't do that, you can't call `atomicInc` on an int*.

I allow conversion to shared. Assume that, and then solve.

Actually, I think I realise the problem. We're not stuck on a failure
of definition, we're stressing over an invalid API!
It's incorrect for atomicInt to receive a shared(int)*, because it
doesn't (and can't) promise thread-safety.
It must receive int*.
The implementation of Atomic which encapsulates access (guards against
unsafe access) to the int will do the proper cast.

> >> One could argue that it should be void free(ref void* p) { /*
> >> ... */ p = null; }
>
> > void *p2 = p;
> > free(p);
> > p2.crash();
>
> That's exactly analogous to what you're proposing: leaking `shared` references while keeping unshared data.

Right, and I'm saying it's completely acceptable and normal.
We don't need to be blocked on this matter.
It's a low-level intrinsic, it isn't a general use API. Can we please
look past it.

void atomicInc(int*); // <- we'll assume this moving forward.

As far as I can tell, this is the ONLY issue you have with the design. Please find another issue.

> > This talk of blunt casts and "making sure everything is good" is all just great, but it doesn't mean  anything interesting with respect to `shared`. It should be interesting even without unsafe casts.
>
> Again, I feel like you didn't see my reply. It's not talk about just blunt casts to make sure everything is good. You either have shared data to begin with, and so can share it freely, or you *know* that you can share this particular piece of data, even if it itself isn't marked `shared`, and you *assert* that by deliberately casting. *You* know, *you* cast, not "some function expects you to know, and just auto-casts".

Sorry, I can't understand this paragraph...

> >> You're missing the point, again. You have an int. You pass a pointer to it to some API that takes an int*. You continue to use your int as just an int.
>
> > You have written an invalid program. I can think of an infinite number of ways to write an invalid program.
>
> No, I have not. I didn't make any promises that my data was shared, and I wasn't expecting it to be treated as such. I didn't even author that API. The other guy (the API) made wrong assumptions. Don't you see the difference here?

I do not. I see it black-and-white, it you write `shared` on a method,
you made a promise to do threadsafety.
You must deliver on that promise. If you do not deliver that promise,
the program is invalid.
If you *can not* deliver on that promise, then you can not attribute
the method shared (or receive a shared argument).

> As I stated previously, there's no difference between Atomic!int and free functions operating on shared int* (or ref shared int). Struct methods are sugared versions of those free functions, *nothing more*. That's why we have UFCS.

I'm not sure what you're saying. Atomic encapsulates guaranteed threadsafe access to an int, whereas shared(int)* doesn't and can't. Any threadsafe API that receives an int will necessarily receive an Atomic!int, unless it's deliberately doing un-safety.

> > That's basic common sense. You don't pass a pointer to a function if you don't know what it does with the pointer!
>
> I should know what the function does with the pointer from it's
> signature. Now, currently in D that very blurry. *Hopefully* with
> 'scope', DIP25 and DIP1000 this becomes more common. But that's
> at least what we should strive for.
> If a function takes `shared`, I better be sure I'm giving it
> `shared`.

You don't care, because you are certain that it can't do anything that's unsafe with your object.

> >> The point is: the caller of
> >> some API *must* assert that they indeed pass shared data. It's
> >> insufficient for the API alone to "promise" taking shared data.
> >> That's the difference with promotion to `const`.
>
> > The caller doesn't care if it's true that the callee can't do
> > anything with it that's unsafe anyway.
> > We effect that state by removing all non-threadsafe access.
>
> You allow non-threadsafe access with implicit cast, not remove it.

Why do you keep saying this?
Item #1: shared instances are completely inaccessible except for
threadsafe methods!

I **ONLY** allow threadsafe access by allowing implicit conversion.

> This broken record is getting very tiresome...

It is. I don't know how to be clearer than I have been.

> Let me ask you
> this once again: *why* are you so bent on this implicit
> conversion from mutable to shared? So far the only reason I've
> seen is to just avoid writing additional methods that forward to
> `shared` methods.

Because users of an API shouldn't have to riddle their code with
manual casts to call perfectly safe functions.
That would be a plain-as-day indicator that the design is wrong.

I need a some real examples of how the implicit cast violates threadsafety...
We've identified the issue with atomicInc(), and it's the only thing
you're stuck on, and you've never picked any general holes in the
design. Please find a legitimate hole in the design...

> Most of your casts will be the other way around, will have to be explicit and there's nothing that can be done about that. You'll only have mutable->shared casts in few select cases, exactly because they're corner cases where you *need* to make the decision clear.

If shared means "threadsafe or no access", then you don't need to make
the decision clear... you don't need to make the decision at all. It's
safe by definition.
I'm trying to create a situation where users are not concerned that
they were able to use an API correctly unless they're *implementing*
threadsafety; that's when they need to think, and any requirement for
un-safety and casting should be factored into that space, not the
calling space.

On Wednesday, 17 October 2018 at 23:12:48 UTC, Manu wrote:
> On Wed, Oct 17, 2018 at 2:15 PM Stanislav Blinov via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>
>> On Wednesday, 17 October 2018 at 19:25:33 UTC, Manu wrote:
>> > On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>> >>
>> >> On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:
>> >>
>> >> > I've said this a bunch of times, there are 2 rules:
>> >> > 1. shared inhibits read and write access to members
>> >> > 2. `shared` methods must be threadsafe
>> >> >
>> >> >>From there, shared becomes interesting and useful.
>> >>
>> >> Oh God...
>> >>
>> >> void atomicInc(shared int* i) { /* ... */ }
>> >>
>> >> Now what? There are no "methods" for ints, only UFCS. Those functions can be as safe as you like, but if you allow implicit promotion of int* to shared int*, you *allow implicit races*.
>> >
>> > This function is effectively an intrinsic. It's unsafe by definition.
>>
>> Only if implicit conversion is allowed. If it isn't, that's likely @trusted, and this:
>>
>> void atomicInc(ref shared int);
>>
>> can even be @safe.
>
> In this case, with respect to the context (a single int) atomicInc()
> is ALWAYS safe, even with implicit conversion. You can atomicInc() a
> thread-local int perfectly safely.
>
>> > It's a tool for implementing threadsafe machinery.
>> > No user can just start doing atomic operations on random ints
>> > and say
>> > "it's threadsafe", you must encapsulate the threadsafe
>> > functionality
>> > into some sort of object that aggregates all concerns and
>> > presents an
>> > intellectually sound api.
>>
>> Threadsafety starts and ends with the programmer. By your logic *all* functions operating on `shared` are unsafe then. As far as compiler is concerned, there would be no difference between these two:
>>
>> struct S {}
>> void atomicInc(ref shared S);
>>
>> and
>>
>> struct S { void atomicInc() shared { /* ... */ } }
>>
>> The signatures of those two functions are exactly the same. How is that different from a function taking a shared int pointer or reference?
>
> It's not, atomicInc() of an int is always safe with respect to the int itself.
> You can call atomicInc() on an unshared int and it's perfectly fine,
> but now you need to consider context, and that's a problem for the
> design of the higher-level scope.
>
> To maintain thread-safety, the int in question must be appropriately contained.
>
> The problem is that the same as the example I presented before, which I'll repeat:
>
> struct InvalidProgram
> {
>   int x;
>   void fun() { ++x; }
>   void gun() shared { atomicInc(&x); }
> }
>
> The method gun() (and therefore the whole object) is NOT threadsafe by
> my definition, because fun() violates the threadsafety of gun().
> The situation applies equally here that:
> int x;
> atomicInc(&x);
> ++x; // <- by my definition, this 'API' (increment an int) violates
> the threadsafety of atomicInc(), and atomicInc() is therefore not
> threadsafe.
>
> `int` doesn't present a threadsafe API, so int is by definition, NOT threadsafe. atomicInc() should be @system, and not @trusted.
>
> If you intend to share an int, use Atomic!int, because it has a threadsafe API.
> atomicInc(shared int*) is effectively just an unsafe intrinsic, and
> its only use is at ground-level implementation of threadsafe
> machinery, like malloc() and free().
>
>> > Let me try one:
>> >
>> > void free(void*) { ... }
>> >
>> > Now what? I might have dangling pointers... it's a catastrophe!
>>
>> One could argue that it should be void free(ref void* p) { /* ...
>> */ p = null; }
>
> void *p2 = p;
> free(p);
> p2.crash();
>
>> As a matter of fact, in my own allocators memory blocks allocated by them are passed by value and are non-copyable, they're not just void[] as in std.experimental.allocator. One must 'move' them to pass ownership, and that includes deallocation. But that's another story altogether.
>
> Right, now you're talking about move semantics to implement transfer of ownership... you might recall I was arguing this exact case to express transferring ownership of objects between threads earlier. This talk of blunt casts and "making sure everything is good" is all just great, but it doesn't mean anything interesting with respect to `shared`. It should be interesting even without unsafe casts.
>
>> > It's essentially the same argument.
>> > This isn't a function that professes to do something that
>> > people might
>> > misunderstand and try to use in an unsafe way, it's a low-level
>> > implementation device, which is used to build larger *useful*
>> > constructs.
>>
>> You're missing the point, again. You have an int. You pass a pointer to it to some API that takes an int*. You continue to use your int as just an int.
>
> You have written an invalid program. I can think of an infinite number
> of ways to write an invalid program.
> In this case, don't have an `int`, instead, have an Atomic!int; you
> now guarantee appropriate access, problem solved!
> If you do have an int, don't pass it to other threads at random when
> you don't have any idea what they intend to do with it! That's basic
> common sense. You don't pass a pointer to a function if you don't know
> what it does with the pointer!
>
>> The API changes, and now the function
>> you called previously takes a shared int*. Implicit conversion
>> works, everything compiles, you have a race. Now, that's of
>> course an extremely stupid scenario.
>
> Yes.
>
>> The point is: the caller of
>> some API *must* assert that they indeed pass shared data. It's
>> insufficient for the API alone to "promise" taking shared data.
>> That's the difference with promotion to `const`.
>
> The caller doesn't care if it's true that the callee can't do anything
> with it that's unsafe anyway.
> We effect that state by removing all non-threadsafe access.

Rather than answering all objections in this thread in detail, a mistake I've made before too, I suggest you put up a 3-5 page document somewhere explaining your proposal in detail. You can use the feedback here as a guide on what to explain more and what not to. Be sure to include code samples of how you see everything ultimately working in your external document.

Two mistakes you may be making in writing responses in this thread, which I've made before too:

1. Assuming people read anything more than fragments of what you're writing. My experience suggests people just read bits and pieces quickly, then compose them together in the way that makes most sense to them based on how _past_ technology works.

2. Assuming people have any idea how the underlying implementation of shared and multi-threading works. I know little to nothing about how shared systems work and how that interacts with type systems, which is why I haven't responded to your proposal, but that doesn't stop others from responding who may not know much more.

Neither of these will be solved by writing an external document, but at least Walter and others more seriously interested will benefit from a better-motivated explanation with more detail. I won't read it, ;) but I think we can avoid another long thread like this if you go this route. I'm planning the same approach with the technical topic I raised before, along with working code.

On Wed, Oct 17, 2018 at 6:50 PM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 10/17/18 6:37 PM, Manu wrote:
> > On Wed, Oct 17, 2018 at 12:35 PM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>
> >> On 10/17/18 2:46 PM, Manu wrote:
> >>> On Wed, Oct 17, 2018 at 10:30 AM Steven Schveighoffer via
> >>
> >>>> What the example demonstrates is that while you are trying to disallow implicit casting of a shared pointer to an unshared pointer, you have inadvertently allowed it by leaving behind an unshared pointer that is the same thing.
> >>>
> >>> This doesn't make sense... you're showing a thread-local program. The thread owning the unshared pointer is entitled to the unshared pointer. It can make as many copies at it likes. They are all thread-local.
> >>
> >> It's assumed that shared int pointer can be passed to another thread, right? Do I have to write a full program to demonstrate?
> >
> > And that shared(int)* provides no access. No other thread with that
> > pointer can do anything with it.
>
> So then it's a misnomer -- it's not really shared, because I can't do anything with it.

**EXACTLY**.. this is the key to the entire design that allows for the implicit promotion.

> >>> There's only one owning thread, and you can't violate that without unsafe casts.
> >>
> >> The what is the point of shared? Like why would you share data that NOBODY CAN USE?
> >
> > You can call shared methods. They promise threadsafety.
> > That's a small subset of the program, but that's natural; only a very
> > small subset of the program is safe to be called from a shared
> > context.
>
> All I can see is that a shared method promises to be callable on shared or unshared data. In essence, it promises nothing.

How is that nothing? It promises threadsafety, such that it can be
called on shared data. Normally, you can do NOTHING with shared data
short of unsafe operations.
It is always safe to call a threadsafe function on an unshared thing,
so it should be possible.
`shared` opts into a strong commitment to threadsafety that allows for
a shared thing to interact with it, otherwise, no interaction is
possible.

I don't understand how you can claim it promises nothing; this is an extremely strong promise!

> It's the programmer who must implement the thread safety, and there really is no help at all from the compiler for this.

I hope we can work on this moving forwards. There are probably some
things we can do... but even without compiler support, in reality,
there are probably only a small number of core pieces of tooling, but
most authors of a shared thing will just aggregate core tools and call
through to their API's.
Most shared code will use tools written by experts, and implement
nothing fancy or magical themselves.

> At some level,
> there will be either casts, or intrinsics, both of which are unsafe
> without knowing all the context of the object.

This will tend to be in the core tooling, written by an expert.
The whole point of my design is to remove friction from the USERS of
those tools, such that they can create aggregate functionality safely
without doing anything unsafe.

> In any case, it's simply
> a false guarantee of thread safety, which might as well be a convention
> of "any function which starts with TS_ is supposed to be thread safe".

What makes it a false guarantee? It allows you to have confidence in
the stack built on top of some core tools.
The whole point here is to minimise the number of people writing code
like that. I'm trying to make shared generally useful so that people
don't have to engage with it at a low-level.

If you see an unsafe case in a shared function (beyond the core tooling), then you should immediately be suspicious, and this is a deliberate part of my goal here. The point here is to produce safer and more reliable code by defining access rules that lead to confident threadsafety.

What you describe is what we have now.

> shared in the current form promises one thing and one thing only -- data marked as shared is actually sharable between threads, and data not marked as shared is actually not shared between threads.

Ummm. The current form might *say* that, but you still have full
unregulated access to all members, and it's completely unsafe, with no
attempt to mitigate that.
This is a new definition. Take it for what it's worth. The old
definition is mostly worthless.
I'm trying to make something that's interesting and useful.

> This new regime you are proposing does nothing extra or new, except break that guarantee.

There are heaps of advantages:
1. If I have a shared thing, I know I can't ruin its state by
manipulating it arbitrarily like I can now
2. I am now able to describe threadsafe objects
3. I relieve users from mental strain of trying to understand how to
correctly and safely interact with shared API's, because they can have
confidence that a shared thing can only perform threadsafe activity
4. I'm trying to give shared a simple, meaningful and *useful*
definition, that's easy to understand and communicate. I believe the
only reason people are having trouble, because they are taking it in
contrast, rather than as it is.

> >> At SOME POINT, shared data needs to be readable and writable. Any correct system is going to dictate how that works. It's a good start to make shared data unusable unless you cast. But then to make it implicitly castable from unshared defeats the whole purpose.
> >
> > No. No casting! This is antiquated workflow.. I'm not trying to take
> > it away from you, but it's not an interesting model for the future.
> > `shared` can model more than just that.
> > You can call threadsafe methods. Shared methods explicitly dictate how
> > the system works, and in a very clear and obvious/intuitive way.
> >
> > The implicit cast makes using threadsafe objects more convenient when you only have one, which is extremely common.
>
> The implicit cast means that you have to look at more than just your method. You have to look at the entire module, and figure out all the interactions, to see if the thread safe method actually is thread safe. That's programming by convention, and fully trusting the programmer.

I don't understand... how can the outer context affect the
threadsafety of a properly encapsulated thing?
Back to this trivial example:

struct Atomic(T)
{
  void opUnary(string op : "++")() shared { atomicInc(cast(T*)&val); }
  private T val;
}

This is properly encapsulated... the promise is firm.
If some threadsafe machinery is sufficiently complex that it has a
wide roam and you can't reason about whether the method is actually
threadsafe, then you have no business writing threadsafe machinery;
use a library.
This design, nor the current design, nor any other design can possibly help you.

> I don't think this thread is going anywhere, so I'll just have to wait and see if someone else can explain it better. I'm a firm no on implicit casting from mutable to shared.

You need to take it for an intellectual spin. Show me how it's corrupt rather than just presenting discomfort with the idea in theory. You're addicted to some concepts that you've carried around for a long time. There is no value in requiring casts, they're just a funky smell, and force the user to perform potentially unsafe manual conversions, or interactions that they don't understand.

Implicit conversion is for allowing/encouraging what is safe. Calling threadsafe functions is safe. You should be able to call threadsafe functions.