On Tue, Oct 16, 2018 at 2:25 AM Kagamin via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
> > Current situation where you can arbitrarily access shared
> > members
> > undermines any value it has.
>
> The value of shared is existence of thread-local data that's guaranteed to be not shared, so you don't need to worry about thread-local data being shared, and shared protects that value well.

This isn't really an argument in favour of shared being able to arbitrarily access all its members though. I'm not sure how your points addresses my claim?

this sounds like that old quote "there can not be peace without war".
It's not shared that makes not-shared thread-local by default, it's
just that's how D is defined.
Shared allows you to break that core assumption. Without shared, it
would still be thread-local by default... you just wouldn't have an
escape hatch ;)

> > Assuming this world... how do you use shared?
>
> Unique solution for each case.
>
> > If you write a lock-free queue for instance, and all the
> > methods are
> > `shared` (ie, threadsafe), then under the current rules, you
> > can't
> > interact with the object when it's not shared, and that's fairly
> > useless.
>
> Create it as shared.

Then you can't use it locally.

> > Assuming the rules above: "can't read or write to members", and the understanding that `shared` methods are expected to have threadsafe implementations (because that's the whole point), what are the risks from allowing T* -> shared(T)* conversion?
>
> All data becomes possibly shared, so you can't assume it's unshared, effectively C-style sharing. BTW D supports the latter already.

No data becomes 'possibly shared', because it's all inaccessible.
Only if your object specifies a threadsafe API may some controlled
data become shared... and that's the whole point of writing a
threadsafe object.
If the threadsafe object doesn't support sharing its own data, then
it's not really a threadsafe object.

On Tue, Oct 16, 2018 at 3:20 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 15.10.2018 20:46, Manu wrote:
> >
> > Assuming the rules above: "can't read or write to members", and the understanding that `shared` methods are expected to have threadsafe implementations (because that's the whole point), what are the risks from allowing T* -> shared(T)* conversion?
> >
>
> Unshared becomes useless, and in turn, shared becomes useless. You can't have unshared/shared aliasing.

What aliasing? Please show a reasonable and likely construction of the problem. I've been trying to think of it.

> > All the risks that I think have been identified previously assume that you can arbitrarily modify the data. That's insanity... assume we fix that... I think the promotion actually becomes safe now...?
>
> But useless, because there is no way to ensure thread safety of reads and writes if only one party to the shared state knows about the sharing.

What? I don't understand this sentence.

If a shared method is not threadsafe, then it's an implementation error. A user should expect that a shared method is threadsafe, otherwise it shouldn't be a shared method! Thread-local (ie, normal) methods are for not-threadsafe functionality.

On Tue, Oct 16, 2018 at 6:25 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 16.10.2018 13:04, Dominikus Dittes Scherkl wrote:
> > On Tuesday, 16 October 2018 at 10:15:51 UTC, Timon Gehr wrote:
> >> On 15.10.2018 20:46, Manu wrote:
> >>>
> >>> Assuming the rules above: "can't read or write to members", and the understanding that `shared` methods are expected to have threadsafe implementations (because that's the whole point), what are the risks from allowing T* -> shared(T)* conversion?
> >>>
> >>
> >> Unshared becomes useless, and in turn, shared becomes useless.
> > why is unshared useless?
> > Unshared means you can read an write to it.
> > If you give it to a function that expect something shared,
> > the function you had given it to can't read or write it, so it
> > can't do any harm.
>
> It can do harm to others who hold an unshared alias to the same data and are operating on it concurrently.

Nobody else holds an unshared alias.
If you pass a value as const, you don't fear that it will become mutable.

> > Of course it can handle it threadsave, but as it is local, that is only overhead - reading or changing the value can't do any harm either. I like the idea.
> >
> >> But useless, because there is no way to ensure thread safety of reads and writes if only one party to the shared state knows about the sharing.
> > Of course there is.
>
> Please do enlighten me. You have two processors operating (reading/writing) on the same address space on a modern computer architecture with a weak memory model, and you are using an optimizing compiler. How do you ensure sensible results without cooperation from both of them? (Hint: you don't.)

What? This is a weird statement.
So, you're saying that nobody has successfully written any threadsafe
code, ever... we should stop trying, and we should admit that
threadsafe queues and atomics, and mutexes and stuff all don't exist?

> without cooperation from both of them?

Perhaps this is the key to your statement?
Yes. 'cooperation from both of them' in this case means, they are both
interacting with a threadsafe api, and they are blocked from accessing
members, or any non-threadsafe api.

> > Giving an unshared value to a function that
> > even can handle shared values may create some overhead, but is
> > indeed threadsave.
> >
>
> Yes, if you give it to one function only, that is the case. However, as you may know, concurrency means that there may be multiple functions operating on the data _at the same time_. If one of them operates on the data as if it was not shared, you will run into trouble.

Who's doing this, and how?

> You are arguing as if there was either no concurrency or no mutable aliasing.

If a class has no shared methods, there's no possibility for mutable aliasing.
If the class has shared methods, then the class was carefully designed
to be threadsafe.

On Tue, Oct 16, 2018 at 6:35 AM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 10/16/18 9:25 AM, Steven Schveighoffer wrote:
> > On 10/15/18 2:46 PM, Manu wrote:
>
> >>> From there, it opens up another critical opportunity; T* -> shared(T)*
> >> promotion.
> >> Const would be useless without T* -> const(T)* promotion. Shared
> >> suffers a similar problem.
> >> If you write a lock-free queue for instance, and all the methods are
> >> `shared` (ie, threadsafe), then under the current rules, you can't
> >> interact with the object when it's not shared, and that's fairly
> >> useless.
> >>
>
> Oh, I didn't see this part. Completely agree with Timon on this, no implicit conversions should be allowed.

Why?

> If you want to have a lock-free implementation of something, you can abstract the assignments and reads behind the proper mechanisms anyway, and still avoid locking (casting is not locking).

Sorry, I don't understand what you're saying. Can you clarify?

On 10/16/18 2:10 PM, Manu wrote:
> On Tue, Oct 16, 2018 at 6:35 AM Steven Schveighoffer via Digitalmars-d
> <digitalmars-d@puremagic.com> wrote:
>>
>> On 10/16/18 9:25 AM, Steven Schveighoffer wrote:
>>> On 10/15/18 2:46 PM, Manu wrote:
>>
>>>>>  From there, it opens up another critical opportunity; T* -> shared(T)*
>>>> promotion.
>>>> Const would be useless without T* -> const(T)* promotion. Shared
>>>> suffers a similar problem.
>>>> If you write a lock-free queue for instance, and all the methods are
>>>> `shared` (ie, threadsafe), then under the current rules, you can't
>>>> interact with the object when it's not shared, and that's fairly
>>>> useless.
>>>>
>>
>> Oh, I didn't see this part. Completely agree with Timon on this, no
>> implicit conversions should be allowed.
> 
> Why?

int x;

shared int *p = &x; // allow implicit conversion, currently error

passToOtherThread(p);

useHeavily(&x);

How is this safe? Thread1 is using x without locking, while the other thread has to lock. In order for synchronization to work, both sides have to agree on a synchronization technique and abide by it.

>> If you want to have a lock-free implementation of something, you can
>> abstract the assignments and reads behind the proper mechanisms anyway,
>> and still avoid locking (casting is not locking).
> 
> Sorry, I don't understand what you're saying. Can you clarify?
> 

I'd still mark a lock-free implementation shared, and all its methods shared. shared does not mean you have to lock, just cast away shared. A lock-free container still has to do some special things to make sure it avoids races, and having an "unusable" state aids in enforcing this.

-Steve

On Tue, Oct 16, 2018 at 11:30 AM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 10/16/18 2:10 PM, Manu wrote:
> > On Tue, Oct 16, 2018 at 6:35 AM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>
> >> On 10/16/18 9:25 AM, Steven Schveighoffer wrote:
> >>> On 10/15/18 2:46 PM, Manu wrote:
> >>
> >>>>>  From there, it opens up another critical opportunity; T* -> shared(T)*
> >>>> promotion.
> >>>> Const would be useless without T* -> const(T)* promotion. Shared
> >>>> suffers a similar problem.
> >>>> If you write a lock-free queue for instance, and all the methods are
> >>>> `shared` (ie, threadsafe), then under the current rules, you can't
> >>>> interact with the object when it's not shared, and that's fairly
> >>>> useless.
> >>>>
> >>
> >> Oh, I didn't see this part. Completely agree with Timon on this, no implicit conversions should be allowed.
> >
> > Why?
>
> int x;
>
> shared int *p = &x; // allow implicit conversion, currently error
>
> passToOtherThread(p);
>
> useHeavily(&x);

What does this mean? It can't do anything... that's the whole point here. I think I'm struggling here with people bringing presumptions to the thread. You need to assume the rules I define in the OP for the experiment to work.

> How is this safe?

Because useHeavily() can't read or write to x.

> Thread1 is using x without locking, while the other
> thread has to lock. In order for synchronization to work, both sides
> have to agree on a synchronization technique and abide by it.

Only the owning thread can access x, the shared instance can't access x at all.

If some code somewhere decides it wants to cast-away shared, then it
needs to determine ownership via some other means. It needs to be
confident that the original owner yielded ownership, and any API that
leads to this behaviour would need to be designed in such a way to
encourage correct behaviour.
That's *exactly* how it is now... I haven't changed anything from this
perspective.

> >> If you want to have a lock-free implementation of something, you can abstract the assignments and reads behind the proper mechanisms anyway, and still avoid locking (casting is not locking).
> >
> > Sorry, I don't understand what you're saying. Can you clarify?
> >
>
> I'd still mark a lock-free implementation shared, and all its methods shared. shared does not mean you have to lock, just cast away shared.

Shared *should* mean that the function is threadsafe, and you are safe
to call it from a shared instance.
If a function is not shared, then you MUST cast away shared, and that
implies that you need to use external means to create a context where
you have thread-local ownership of the instance (usually with a
mutex).

You shouldn't need to cast-away shared to make a safe function call.
By casting away shared, you also gain access to all the non-threadsafe
methods and members.
Casting shared away to call a shared method makes a safe access into a
potentially unsafe access.

It's unacceptable to case-away shared. It should be as unacceptable as casting const away. The only situation where it's okay is where you're externally verifying thread-locality by external means, and that's subject to your broader systemic design.

> A lock-free container still has to do some special things to make sure it avoids races, and having an "unusable" state aids in enforcing this.

Can you explain how my proposal doesn't model this very neatly?

On 10/16/18 4:26 PM, Manu wrote:
> On Tue, Oct 16, 2018 at 11:30 AM Steven Schveighoffer via
> Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>
>> On 10/16/18 2:10 PM, Manu wrote:
>>> On Tue, Oct 16, 2018 at 6:35 AM Steven Schveighoffer via Digitalmars-d
>>> <digitalmars-d@puremagic.com> wrote:
>>>>
>>>> On 10/16/18 9:25 AM, Steven Schveighoffer wrote:
>>>>> On 10/15/18 2:46 PM, Manu wrote:
>>>>
>>>>>>>   From there, it opens up another critical opportunity; T* -> shared(T)*
>>>>>> promotion.
>>>>>> Const would be useless without T* -> const(T)* promotion. Shared
>>>>>> suffers a similar problem.
>>>>>> If you write a lock-free queue for instance, and all the methods are
>>>>>> `shared` (ie, threadsafe), then under the current rules, you can't
>>>>>> interact with the object when it's not shared, and that's fairly
>>>>>> useless.
>>>>>>
>>>>
>>>> Oh, I didn't see this part. Completely agree with Timon on this, no
>>>> implicit conversions should be allowed.
>>>
>>> Why?
>>
>> int x;
>>
>> shared int *p = &x; // allow implicit conversion, currently error
>>
>> passToOtherThread(p);
>>
>> useHeavily(&x);
> 
> What does this mean? It can't do anything... that's the whole point here.
> I think I'm struggling here with people bringing presumptions to the
> thread. You need to assume the rules I define in the OP for the
> experiment to work.

OK, I wrote a whole big response to this, and I went and re-quoted the above, and now I think I understand what the point of your statement is.

I'll first say that if you don't want to allow implicit casting of shared to mutable, then you can't allow implicit casting from mutable to shared. Because it's mutable, races can happen.

There is in fact, no difference between:

int *p;
shared int *p2 = p;
int *p3 = cast(int*)p2;

and this:

int *p;
shared int *p2 = p;
int *p3 = p;

So really, the effort to prevent the reverse cast is defeated by allowing the implicit cast.

There is a reason we disallow assigning from mutable to immutable without a cast. Yet, it is done in many cases, because you are sometimes building an immutable object with mutable pieces, and want to cast the final result.

In this case, it's ON YOU to make sure it's correct, and the traditional mechanism for the compiler giving you the responsibility is to require a cast.

-----

OK, so here is where I think I misunderstood your point. When you said a lock-free queue would be unusable if it wasn't shared, I thought you meant it would be unusable if we didn't allow the implicit cast. But I realize now, you meant you should be able to use a lock-free queue without it being actually shared anywhere.

What I say to this is that it doesn't need to be usable. I don't care to use a lock-free queue in a thread-local capacity. I'll just use a normal queue, which is easy to implement, and doesn't have to worry about race conditions or using atomics. A lock free queue is a special thing, very difficult to get right, and only really necessary if you are going to share it. And used for performance reasons!

Why would I want to incur performance penalties when using a lock-free queue in an unshared mode? I would actually expect 2 separate implementations of the primitives, one for shared one for unshared.

What about primitives that would be implemented the same? In that case, the shared method becomes:

auto method() { return (cast(Queue*)&this).method; }

Is this "unusable"? Without a way to say, you can call this on shared or unshared instances, then we need to do it this way.

But I would trust the queue to handle this properly depending on whether it was typed shared or not.

-Steve

On Tuesday, 16 October 2018 at 21:19:26 UTC, Steven Schveighoffer wrote:
> There is in fact, no difference between:
>
> int *p;
> shared int *p2 = p;
> int *p3 = cast(int*)p2;
>
> and this:
>
> int *p;
> shared int *p2 = p;
> int *p3 = p;

If I understand Manu correctly the first should compile, and the second should error, just like if you replaces shared with const in the above.

On Tuesday, 16 October 2018 at 21:19:26 UTC, Steven Schveighoffer wrote:
> OK, so here is where I think I misunderstood your point. When you said a lock-free queue would be unusable if it wasn't shared, I thought you meant it would be unusable if we didn't allow the implicit cast. But I realize now, you meant you should be able to use a lock-free queue without it being actually shared anywhere.
>
> What I say to this is that it doesn't need to be usable. I don't care to use a lock-free queue in a thread-local capacity. I'll just use a normal queue, which is easy to implement, and doesn't have to worry about race conditions or using atomics. A lock free queue is a special thing, very difficult to get right, and only really necessary if you are going to share it. And used for performance reasons!

I think this comes up where the queue was originally shared, you acquired a lock on the thing it is a member of, and you want to continue using it through your exclusive reference.

On Tue, Oct 16, 2018 at 2:20 PM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 10/16/18 4:26 PM, Manu wrote:
> > On Tue, Oct 16, 2018 at 11:30 AM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>
> >> On 10/16/18 2:10 PM, Manu wrote:
> >>> On Tue, Oct 16, 2018 at 6:35 AM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>>>
> >>>> On 10/16/18 9:25 AM, Steven Schveighoffer wrote:
> >>>>> On 10/15/18 2:46 PM, Manu wrote:
> >>>>
> >>>>>>>   From there, it opens up another critical opportunity; T* -> shared(T)*
> >>>>>> promotion.
> >>>>>> Const would be useless without T* -> const(T)* promotion. Shared
> >>>>>> suffers a similar problem.
> >>>>>> If you write a lock-free queue for instance, and all the methods are
> >>>>>> `shared` (ie, threadsafe), then under the current rules, you can't
> >>>>>> interact with the object when it's not shared, and that's fairly
> >>>>>> useless.
> >>>>>>
> >>>>
> >>>> Oh, I didn't see this part. Completely agree with Timon on this, no implicit conversions should be allowed.
> >>>
> >>> Why?
> >>
> >> int x;
> >>
> >> shared int *p = &x; // allow implicit conversion, currently error
> >>
> >> passToOtherThread(p);
> >>
> >> useHeavily(&x);
> >
> > What does this mean? It can't do anything... that's the whole point here. I think I'm struggling here with people bringing presumptions to the thread. You need to assume the rules I define in the OP for the experiment to work.
>
> OK, I wrote a whole big response to this, and I went and re-quoted the above, and now I think I understand what the point of your statement is.
>
> I'll first say that if you don't want to allow implicit casting of shared to mutable,

It's critical that this is not allowed. It's totally unreasonable to
cast from shared to thread-local without synchronisation.
It's as bad as casting away const.

> then you can't allow implicit casting from mutable to shared. Because it's mutable, races can happen.

I don't follow...

> There is in fact, no difference between:
>
> int *p;
> shared int *p2 = p;
> int *p3 = cast(int*)p2;

Totally illegal!! You casted away shared. That's as bad as casting away const.

> and this:
>
> int *p;
> shared int *p2 = p;
> int *p3 = p;

There's nothing wrong with this... I don't understand the point?

> So really, the effort to prevent the reverse cast is defeated by allowing the implicit cast.

Only the caller has the thread-local instance. You can take a
thread-local pointer to a thread-local within the context of a single
thread.
So, it's perfectly valid for `p` and `p3` to exist in a single scope.
`p2` is fine here too... and if that shared pointer were to escape to
another thread, it wouldn't be a threat, because it's not readable or
writable, and you can't make it back into a thread-local pointer
without carefully/deliberately deployed machinery.

> There is a reason we disallow assigning from mutable to immutable without a cast. Yet, it is done in many cases, because you are sometimes building an immutable object with mutable pieces, and want to cast the final result.

I don't think analogy to immutable has a place in this discussion, or
at least, I don't understand the relevance...
I think the reasonable analogy is const.

> In this case, it's ON YOU to make sure it's correct, and the traditional mechanism for the compiler giving you the responsibility is to require a cast.

I think what you're talking about are behaviours relating to casting
shared *away*, and that's some next-level shit. Handling in that case
is no different to the way it exists today. You must guarantee that
the pointer you possess becomes thread-local before casting it to a
thread-local pointer.
In my application framework, I will never cast shared away under my
proposed design. We don't have any such global locks.

> -----
>
> OK, so here is where I think I misunderstood your point. When you said a lock-free queue would be unusable if it wasn't shared, I thought you meant it would be unusable if we didn't allow the implicit cast. But I realize now, you meant you should be able to use a lock-free queue without it being actually shared anywhere.

Right, a lock-free queue is a threadsafe object, and it's methods work
whether the queue is shared or not.
The methods are attributed shared because they can be called on shared
instances... but they can ALSO be called from a thread-local instance,
and under my suggested promotion rules, it's fine for the this-pointer
to promote to shared to make the call.

> What I say to this is that it doesn't need to be usable. I don't care to use a lock-free queue in a thread-local capacity. I'll just use a normal queue, which is easy to implement, and doesn't have to worry about race conditions or using atomics. A lock free queue is a special thing, very difficult to get right, and only really necessary if you are going to share it. And used for performance reasons!

I'm more interested in the object that has that lock-free queue as a
member... it is probably a mostly thread-local object, but may have a
couple of shared methods.
I have a whole lot of objects which have 3 tiers of API access; the
thread-local part, the threadsafe part, and the const part. Just as a
mutable instance can call a const method, there's no reason a
thread-local instance can't call a threadsafe method.

You can ask 'why', and I can't give any more satisfactory answer than
"you can call a const method from a mutable object", this is
commonsense, and should be possible.
We want to call shared methods from threadlocal objects all the time.
It's possible, and it's safe... there's no reason we shouldn't be able
to.

> Why would I want to incur performance penalties when using a lock-free queue in an unshared mode? I would actually expect 2 separate implementations of the primitives, one for shared one for unshared.

Overloading for shared and unshared is possible, and may be desirable
in many cases.
There are also many cases where the code duplication and tech-debt
does not carry its weight. It should not be required, because it's not
technically required.

> What about primitives that would be implemented the same? In that case, the shared method becomes:
>
> auto method() { return (cast(Queue*)&this).method; }
>
> Is this "unusable"? Without a way to say, you can call this on shared or unshared instances, then we need to do it this way.

I don't understand here...?
I'm saying, we *don't* need a way to say "you can call on shared or
unshared instances*, because that's always safe to do. If it wasn't
safe to call a const method with a mutable instance, something is
terrible wrong; same applies here, it is always safe to call a
threadsafe method with a thread-local instance. If that's not true,
the method is objectively not threadsafe.

> But I would trust the queue to handle this properly depending on whether it was typed shared or not.

The queue is less interesting than the object that aggregates the queue.