On 10/18/18 2:24 PM, Manu wrote:
> I understand your argument, and I used to think this too... but I
> concluded differently for 1 simple reason: usability.

You have not demonstrated why your proposal is usable, and the proposal to simply make shared not accessible while NOT introducing implicit conversion is somehow not usable.

I find quite the opposite -- the implicit conversion introduces more pitfalls and less guarantees from the compiler.

> I have demonstrated these usability considerations in production. I am
> confident it's the right balance.

Are these considerations the list below, or are they something else? If so, can you list them?

> I propose:
>   1. Normal people don't write thread-safety, a very small number of
> unusual people do this. I feel very good about biasing 100% of the
> cognitive load INSIDE the shared method. This means the expert, and
> ONLY the expert, must make decisions about thread-safety
> implementation.

Thread safety is not easy. But it's also not generic.

In terms of low-level things like atomics and lock-free implementations, those ARE generic and SHOULD only be written by experts. But other than that, you can't know how someone has designed all the conditions in their code.

For example, you can have an expert write mutex locks and semaphores. But they can't tell you the proper order to lock different objects to ensure there's no deadlock. That's application specific.

>   2. Implicit conversion allows users to safely interact with safe
> things without doing unsafe casts. I think it's a complete design fail
> if you expect any user anywhere to perform an unsafe cast to call a
> perfectly thread-safe function. The user might not properly understand
> their obligations.

I also do not expect anyone to perform unsafe casts in normal use. I expect them to use more generic well-written types in a shared-object library. Casting should be very rare.

>   3. The practical result of the above is, any complexity relating to
> safety is completely owned by the threadsafe author, and not cascaded
> to the user. You can't expect users to understand, and make correct
> decisions about threadsafety. Safety should be default position.

I think these are great rules, and none are broken by keeping the explicit cast requirement in place.

> I recognise the potential loss of an unsafe optimised thread-local path.
> 1. This truly isn't a big deal. If this is really hurting you, you
> will notice on the profiler, and deploy a thread-exclusive path
> assuming the context supports it.

This is a mischaracterization. The thread-local path is perfectly safe because only one thread can be accessing the data. That's why it's thread-local and not shared.

> 2. I will trade that for confidence in safe interaction every day of
> the week. Safety is the right default position here.

You can be confident that any shared data is properly synchronized via the API provided. No confidence should be lost here.

> 2. You just need to make the unsafe thread-exclusive variant explicit, eg:

It is explicit, the thread-exclusive variant is not marked shared, and cannot be called on data that is actually shared and needs synchronization.

> 
>> struct ThreadSafe
>> {
>>      private int x;
>>      void unsafeIncrement() // <- make it explicit
>>      {
>>         ++x; // User has asserted that no sharing is possible, no reason to use atomics
>>      }
>>      void increment() shared
>>      {
>>         atomicIncrement(&x); // object may be shared
>>      }
>> }

This is more design by convention.

> 
> I think this is quiet a reasonable and clearly documented compromise.
> I think absolutely-reliably-threadsafe-by-default is the right default
> position. And if you want to accept unsafe operations for optimsation
> circumstances, then you're welcome to deploy that in your code as you
> see fit.

All thread-local operations are thread-safe by default, because there can be only one thread using it. That is the beauty of the current regime, regardless of how broken shared is -- unshared is solid. We shouldn't want to break that guarantee.

> If the machinery is not a library for distribution and local to your
> application, and you know for certain that your context is such that
> thread-local and shared are mutually exclusive, then you're free to
> make the unshared overload not-threadsafe; you can do this because you
> know your application context.
> You just shouldn't make widely distributed tooling this way.

I can make widely distributed tooling that does both shared and unshared versions of the code, and ALL are thread safe. No choices are necessary, no compromise on performance, and no design by convention.

> I will indeed do this myself in some cases, because I know those facts
> about my application.
> But I wouldn't compromise the default design of shared for this
> optimisation potential... deliberately deployed optimisation is okay
> to be unsafe when taken in context.
> 

Except it's perfectly thread safe to use data without synchronization in one thread -- which is supported by having unshared data. Unshared means only one thread. In your proposal, anything can be seen from one or more threads.

-Steve

On 10/18/18 2:55 PM, Manu wrote:
> On Thu, Oct 18, 2018 at 7:20 AM Steven Schveighoffer via Digitalmars-d
> <digitalmars-d@puremagic.com> wrote:
>>
>> On 10/18/18 10:11 AM, Simen Kjærås wrote:
>>> On Thursday, 18 October 2018 at 13:35:22 UTC, Steven Schveighoffer wrote:
>>>> struct ThreadSafe
>>>> {
>>>>     private int x;
>>>>     void increment()
>>>>     {
>>>>        ++x; // I know this is not shared, so no reason to use atomics
>>>>     }
>>>>     void increment() shared
>>>>     {
>>>>        atomicIncrement(&x); // use atomics, to avoid races
>>>>     }
>>>> }
>>>
>>> But this isn't thread-safe, for the exact reasons described elsewhere in
>>> this thread (and in fact, incorrectly leveled at Manu's proposal).
>>> Someone could write this code:
>>>
>>> void foo() {
>>>       ThreadSafe* a = new ThreadSafe();
>>>       shareAllOver(a);
>>
>> Error: cannot call function shareAllOver(shared(ThreadSafe) *) with type
>> ThreadSafe *
> 
> And here you expect a user to perform an unsafe-cast (which they may
> not understand), and we have no language semantics to enforce the
> transfer of ownership. How do you assure that the user yields the
> thread-local instance?

No, I expect them to do:

auto a = new shared(ThreadSafe)();

> I think requiring the cast is un-principled in every way that D values.

No cast is required. If you have shared data, it's shared. If you have thread local data, it's unshared. Allocate the data the way you expect to use it.

It's only if you intend to turn unshared data into shared data where you need an unsafe cast.

It's not even as difficult as immutable, because you can still modify shared data. For instance, the shared constructor doesn't have to have special rules about initialization, it can just assume shared from the beginning.

-Steve

On 10/18/18 2:59 PM, Manu wrote:
> On Thu, Oct 18, 2018 at 7:20 AM Steven Schveighoffer via Digitalmars-d
> <digitalmars-d@puremagic.com> wrote:
>>
>> On 10/18/18 10:11 AM, Simen Kjærås wrote:
>>>       a.increment(); // unsafe, non-shared method call
>>> }
>>>
>>> When a.increment() is being called, you have no idea if anyone else is
>>> using the shared interface.
>>
>> I do, because unless you have cast the type to shared, I'm certain there
>> is only thread-local aliasing to it.
> 
> No, you can never be sure. Your assumption depends on the *user*
> engaging in an unsafe operation (the cast), and correctly perform a
> conventional act; they must correctly the safely transfer ownership.

Not at all. No transfer of ownership is needed, no cast is needed. If you want to share something declare it shared.

> My proposal puts all requirements on the author, not the user. I think
> this is a much more trustworthy relationship, and in terms of
> cognitive load, author:users is a 1:many relationship, and I place the
> load on the '1', not the 'many.

Sure, but we can create a system today where smart people make objects that do the right thing without compiler help. We don't need to break the guarantees of shared to do it.

-Steve

On 10/18/18 2:42 PM, Stanislav Blinov wrote:
> On Thursday, 18 October 2018 at 18:26:27 UTC, Steven Schveighoffer wrote:
>> On 10/18/18 1:47 PM, Stanislav Blinov wrote:
>>> On Thursday, 18 October 2018 at 17:17:37 UTC, Atila Neves wrote:
>>>> On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
>>>>> 1. shared should behave exactly like const, except in addition to inhibiting write access, it also inhibits read access.
>>>>
>>>> How is this significantly different from now?
>>>>
>>>> -----------------
>>>> shared int i;
>>>> ++i;
>>>>
>>>> Error: read-modify-write operations are not allowed for shared variables. Use core.atomic.atomicOp!"+="(i, 1) instead.
>>>> -----------------
>>>>
>>>> There's not much one can do to modify a shared value as it is.
>>>
>>> i = 1;
>>> int x = i;
>>> shared int y = i;
>>
>> This should be fine, y is not shared when being created.
> 
> 'y' isn't, but 'i' is. It's fine on amd64, but that's incidental.

OH, I didn't even notice that `i` didn't have a type, so it was a continuation of the original example! I read it as declaring y as shared and assigning it to a thread-local (which it isn't actually).

My bad.

-Steve

On Thursday, 18 October 2018 at 19:04:58 UTC, Erik van Velzen wrote:
> On Thursday, 18 October 2018 at 17:47:29 UTC, Stanislav Blinov wrote:
>> On Thursday, 18 October 2018 at 17:17:37 UTC, Atila Neves wrote:
>>> On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
>>>> Assuming the rules above: "can't read or write to members", and the understanding that `shared` methods are expected to have threadsafe implementations (because that's the whole point), what are the risks from allowing T* -> shared(T)* conversion?
>>>
>>> int i;
>>> tid.send(&i);
>>> ++i;  // oops, data race
>>
>> Doesn't work. No matter what you show Manu or Simen here they think it's just a bad contrived example. You can't sway them by the fact that the compiler currently *prevents* this from happening.
>
> Manu said clearly that the receiving thread won't be able to read or write the pointer.

Yes it will, by casting `shared` away. *Just like* his proposed "wrap everything into" struct will. There's exactly no difference.

> Because int or int* does not have threadsafe member functions.

int doesn't have any member functions. Or it can have as many as you like per UFCS. Same goes for structs. Because "methods" are just free functions in disguise, so that whole distinction in Manu's proposal is a weaksauce convention at best.

> You can still disagree on the merits, but so far it has been demonstrated as a sound idea.

No, it hasn't been.

On Thursday, 18 October 2018 at 19:26:39 UTC, Stanislav Blinov wrote:
> On Thursday, 18 October 2018 at 19:04:58 UTC, Erik van Velzen wrote:
>> On Thursday, 18 October 2018 at 17:47:29 UTC, Stanislav Blinov wrote:
>>>
>>> Doesn't work. No matter what you show Manu or Simen here they think it's just a bad contrived example. You can't sway them by the fact that the compiler currently *prevents* this from happening.
>>
>> Manu said clearly that the receiving thread won't be able to read or write the pointer.
>
> Yes it will, by casting `shared` away. *Just like* his proposed "wrap everything into" struct will. There's exactly no difference.
>

Casting is inherently unsafe. Or at least, there's no threadsafe guarantee.

>> Because int or int* does not have threadsafe member functions.
>
> int doesn't have any member functions. Or it can have as many as you like per UFCS. Same goes for structs. Because "methods" are just free functions in disguise, so that whole distinction in Manu's proposal is a weaksauce convention at best.
>
>> You can still disagree on the merits, but so far it has been demonstrated as a sound idea.
>
> No, it hasn't been.

I think you are missing the wider point. I can write thread-unsafe code *right now*, no casts required. Just put shared at the declaration. The proposal would actually give some guarantees.

On Thursday, 18 October 2018 at 19:51:17 UTC, Erik van Velzen wrote:
> On Thursday, 18 October 2018 at 19:26:39 UTC, Stanislav Blinov

>>> Manu said clearly that the receiving thread won't be able to read or write the pointer.
>>
>> Yes it will, by casting `shared` away. *Just like* his proposed "wrap everything into" struct will. There's exactly no difference.
>>

> Casting is inherently unsafe. Or at least, there's no threadsafe guarantee.

So? That's the only way to implement required low-level access, especially if we imagine that the part of Manu's proposal about disabling reads and writes on `shared` values is a given. It's the only way to implement Manu's Atomic!int, or at least operation it requires, for example.

>>> You can still disagree on the merits, but so far it has been demonstrated as a sound idea.
>>
>> No, it hasn't been.
>
> I think you are missing the wider point. I can write thread-unsafe code *right now*, no casts required. Just put shared at the declaration. The proposal would actually give some guarantees.

No, I think you are missing the wider point. You can write thread-unsafe code regardless of using `shared`, and regardless of it's implementation details. Allowing *implicit automatic promotion* of *mutable thread-local* data to shared will allow you to write even more thread-unsafe code, not less.

The solid part of the proposal is about disabling reads and writes. The rest is pure convention: write structs instead of functions, and somehow (???) benefit from a totally unsafe implicit cast.

On Thu, Oct 18, 2018 at 12:10 PM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 10/18/18 2:24 PM, Manu wrote:
> > I understand your argument, and I used to think this too... but I concluded differently for 1 simple reason: usability.
>
> You have not demonstrated why your proposal is usable, and the proposal to simply make shared not accessible while NOT introducing implicit conversion is somehow not usable.
>
> I find quite the opposite -- the implicit conversion introduces more pitfalls and less guarantees from the compiler.

I don't think it introduces *more*, I think it's the same number of
compiler guarantees, but it rearranges them into what I feel are a
more satisfactory and reliable configuration.
Ie, I rearrange such that the compiler guarantees are applicable to
the 'many' case rather than the '1' case, and from that perspective,
it means the compiler guarantees are more widely deployed.

> > I have demonstrated these usability considerations in production. I am confident it's the right balance.
>
> Are these considerations the list below, or are they something else? If so, can you list them?
>
> > I propose:
> >   1. Normal people don't write thread-safety, a very small number of
> > unusual people do this. I feel very good about biasing 100% of the
> > cognitive load INSIDE the shared method. This means the expert, and
> > ONLY the expert, must make decisions about thread-safety
> > implementation.
>
> Thread safety is not easy. But it's also not generic.

They are generic in lots of instances. A lock-free queue object is a
generic container object which can be deployed safely.
A threadsafe state-machine implementation which implements reliable
and valid state transitions using atomics is generic.
Most of the threadsafe tooling I've ever seen expose at the
user-facing level is absolutely generic, and can be packaged as a safe
and user-friendly abstraction.

> In terms of low-level things like atomics and lock-free implementations, those ARE generic and SHOULD only be written by experts. But other than that, you can't know how someone has designed all the conditions in their code.

That guy is implementing the machinery, and he is the best possible person to validate that he delivered on his promises. Nobody should have to perform un-safety to interact with his promises. There is greater chance of user error than expert failure (users are, by definition, more numerous in number, and almost certainly less qualified).

> For example, you can have an expert write mutex locks and semaphores. But they can't tell you the proper order to lock different objects to ensure there's no deadlock. That's application specific.

A mutex-style API has an element of un-safety by definition. My
proposal doesn't affect lock and cast-away workflows.
That workflow remains the same, and depends on unsafe interactions,
and I don't think it's possible to arrange that any other way.
What I'm trying to do is express another form of safe interaction
tools with threadsafe devices, and in my work, I would use those
exclusively. I have no use or desire for unsafe lock-and-cast
workflows in our ecosystem.
I'm trying to add a new possibility for expressing threadsafety that
doesn't exist with strong guarantees today.

> >   2. Implicit conversion allows users to safely interact with safe
> > things without doing unsafe casts. I think it's a complete design fail
> > if you expect any user anywhere to perform an unsafe cast to call a
> > perfectly thread-safe function. The user might not properly understand
> > their obligations.
>
> I also do not expect anyone to perform unsafe casts in normal use. I expect them to use more generic well-written types in a shared-object library. Casting should be very rare.

You're resistant to implicit conversion to shared. Casting to shared
is unsafe, and depending on them to yield thread-local ownership is a
'hope' at best; your worldview depends on users performing unsafe
interactions with otherwise safe (threadsafe) API's.
I'm trying to reposition away from that world into a safe-by-default place.

> >   3. The practical result of the above is, any complexity relating to
> > safety is completely owned by the threadsafe author, and not cascaded
> > to the user. You can't expect users to understand, and make correct
> > decisions about threadsafety. Safety should be default position.
>
> I think these are great rules, and none are broken by keeping the explicit cast requirement in place.

They are expressly broken whenever anyone has to cast to shared. That
is unsafe, and they're expected to yield the thread-local ownership by
convention.
That is the cancer at the core of my worldview, which I'm trying to
factor away. I think my proposal delivers on that very elegantly.
There will be *no casts* anywhere, outside of the low-level
implementation methods written by the expert.
I am aiming for safe-by-default. If you want to step outside that
place, you do so deliberate, and carefully, and it's easy to search
for.

> > I recognise the potential loss of an unsafe optimised thread-local path.
> > 1. This truly isn't a big deal. If this is really hurting you, you
> > will notice on the profiler, and deploy a thread-exclusive path
> > assuming the context supports it.
>
> This is a mischaracterization. The thread-local path is perfectly safe because only one thread can be accessing the data. That's why it's thread-local and not shared.

It's not safe. There may be a thread-local instance at any time,
because we have no mechanism to concretely transfer ownership. You
rely on unsafe cast and convention to yield ownership to perform the
transition.
I'm saying, I don't find that acceptable, and I'm designing for that
reality, rather than wishful thinking.

> > 2. I will trade that for confidence in safe interaction every day of the week. Safety is the right default position here.
>
> You can be confident that any shared data is properly synchronized via the API provided. No confidence should be lost here.

But you can't safely call a threadsafe method, and you can't transition TL -> shared data. My proposal addresses those issues.

> > 2. You just need to make the unsafe thread-exclusive variant explicit, eg:
>
> It is explicit, the thread-exclusive variant is not marked shared, and cannot be called on data that is actually shared and needs synchronization.

It's defeated by the other considerations though. Implicit conversion is the only way to allow data to become shared safely, and the design works elegantly.

> >> struct ThreadSafe
> >> {
> >>      private int x;
> >>      void unsafeIncrement() // <- make it explicit
> >>      {
> >>         ++x; // User has asserted that no sharing is possible, no reason to use atomics
> >>      }
> >>      void increment() shared
> >>      {
> >>         atomicIncrement(&x); // object may be shared
> >>      }
> >> }
>
> This is more design by convention.

You want to do an unsafe thing. You need to prescribe convention when
you want to do unsafe things.
I don't recommend this, I'm telling you that you can do it in your
case of desired optimisation.

> > I think this is quiet a reasonable and clearly documented compromise. I think absolutely-reliably-threadsafe-by-default is the right default position. And if you want to accept unsafe operations for optimsation circumstances, then you're welcome to deploy that in your code as you see fit.
>
> All thread-local operations are thread-safe by default, because there can be only one thread using it. That is the beauty of the current regime, regardless of how broken shared is -- unshared is solid. We shouldn't want to break that guarantee.

Right, but you just said it; shared is useless by extension. I'm
trying to reconcile shared with the current design in such a way to
express a useful concept.
I'm not undermining how thread-local is threadsafe by default, that
promise still exists unchanged.
What I'm saying is, promise of threadsafety must be true with respect
to the threadlocal implementation, that is all. I think it's a
reasonable definition for threadsafety, and the value is evident; you
don't need to perform unsafe operations and convention to do
interaction with threadsafe machinery (as it should be, because it is
*threadsafe*).

> > If the machinery is not a library for distribution and local to your
> > application, and you know for certain that your context is such that
> > thread-local and shared are mutually exclusive, then you're free to
> > make the unshared overload not-threadsafe; you can do this because you
> > know your application context.
> > You just shouldn't make widely distributed tooling this way.
>
> I can make widely distributed tooling that does both shared and unshared versions of the code, and ALL are thread safe. No choices are necessary, no compromise on performance, and no design by convention.

You provide no path to make an unshared thing shared. I don't think
the language has any tools to express this; we can't express an
ownership transfer.
The mutually-exclusive shared-ness design fails here. I'm designing for reality.

> > I will indeed do this myself in some cases, because I know those facts
> > about my application.
> > But I wouldn't compromise the default design of shared for this
> > optimisation potential... deliberately deployed optimisation is okay
> > to be unsafe when taken in context.
> >
>
> Except it's perfectly thread safe to use data without synchronization in one thread -- which is supported by having unshared data. Unshared means only one thread. In your proposal, anything can be seen from one or more threads.

Right, but as I've tried to demonstrate, shared can't exist safely
with this construction, no transition is possible.
I consider that a non-starter, and my design addresses that reality.

On Thu, Oct 18, 2018 at 12:15 PM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 10/18/18 2:55 PM, Manu wrote:
> > On Thu, Oct 18, 2018 at 7:20 AM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>
> >> On 10/18/18 10:11 AM, Simen Kjærås wrote:
> >>> On Thursday, 18 October 2018 at 13:35:22 UTC, Steven Schveighoffer wrote:
> >>>> struct ThreadSafe
> >>>> {
> >>>>     private int x;
> >>>>     void increment()
> >>>>     {
> >>>>        ++x; // I know this is not shared, so no reason to use atomics
> >>>>     }
> >>>>     void increment() shared
> >>>>     {
> >>>>        atomicIncrement(&x); // use atomics, to avoid races
> >>>>     }
> >>>> }
> >>>
> >>> But this isn't thread-safe, for the exact reasons described elsewhere in this thread (and in fact, incorrectly leveled at Manu's proposal). Someone could write this code:
> >>>
> >>> void foo() {
> >>>       ThreadSafe* a = new ThreadSafe();
> >>>       shareAllOver(a);
> >>
> >> Error: cannot call function shareAllOver(shared(ThreadSafe) *) with type
> >> ThreadSafe *
> >
> > And here you expect a user to perform an unsafe-cast (which they may not understand), and we have no language semantics to enforce the transfer of ownership. How do you assure that the user yields the thread-local instance?
>
> No, I expect them to do:
>
> auto a = new shared(ThreadSafe)();

I don't have any use for this design in my application.
I can't use the model you prescribe, at all.

> > I think requiring the cast is un-principled in every way that D values.
>
> No cast is required. If you have shared data, it's shared. If you have thread local data, it's unshared. Allocate the data the way you expect to use it.

All data is thread-local, and occasionally becomes shared during periods. I can't make use of the model you describe.

My proposal is more permissive, and allows a wider range of application designs. What are the disadvantages?

> It's only if you intend to turn unshared data into shared data where you need an unsafe cast.

It's unnecessary though, because threadsafe functions are threadsafe! You're pointlessly forcing un-safety. Why would I prefer a design that forces unsafe interactions to perform safe operations?

> It's not even as difficult as immutable, because you can still modify shared data. For instance, the shared constructor doesn't have to have special rules about initialization, it can just assume shared from the beginning.

Your design us immutable, mine is const.
Tell me, how many occurrences of 'immutable' can you find in your
software? ... how about const?
Which is more universally useful? If you had to choose one or the
other, which one could you live without?

On Thu, Oct 18, 2018 at 1:10 PM Stanislav Blinov via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On Thursday, 18 October 2018 at 19:51:17 UTC, Erik van Velzen wrote:
> > On Thursday, 18 October 2018 at 19:26:39 UTC, Stanislav Blinov
>
> >>> Manu said clearly that the receiving thread won't be able to read or write the pointer.
> >>
> >> Yes it will, by casting `shared` away. *Just like* his proposed "wrap everything into" struct will. There's exactly no difference.
> >>
>
> > Casting is inherently unsafe. Or at least, there's no threadsafe guarantee.
>
> So? That's the only way to implement required low-level access, especially if we imagine that the part of Manu's proposal about disabling reads and writes on `shared` values is a given. It's the only way to implement Manu's Atomic!int, or at least operation it requires, for example.

@trusted code exists, and it's the foundation of the @safe stack. I think you're just trying to be obtuse at this point.