On 16.10.2018 20:07, Manu wrote:
> On Tue, Oct 16, 2018 at 6:25 AM Timon Gehr via Digitalmars-d
> <digitalmars-d@puremagic.com> wrote:
>>
>> On 16.10.2018 13:04, Dominikus Dittes Scherkl wrote:
>>> On Tuesday, 16 October 2018 at 10:15:51 UTC, Timon Gehr wrote:
>>>> On 15.10.2018 20:46, Manu wrote:
>>>>>
>>>>> Assuming the rules above: "can't read or write to members", and the
>>>>> understanding that `shared` methods are expected to have threadsafe
>>>>> implementations (because that's the whole point), what are the risks
>>>>> from allowing T* -> shared(T)* conversion?
>>>>>
>>>>
>>>> Unshared becomes useless, and in turn, shared becomes useless.
>>> why is unshared useless?
>>> Unshared means you can read an write to it.
>>> If you give it to a function that expect something shared,
>>> the function you had given it to can't read or write it, so it
>>> can't do any harm.
>>
>> It can do harm to others who hold an unshared alias to the same data and
>> are operating on it concurrently.
> 
> Nobody else holds an unshared alias.

How so? If you allow implicit conversions from unshared to shared, then you immediately get this situation.

> If you pass a value as const, you don't fear that it will become mutable.
> ...

No, but as I already explained last time, mutable -> const is not at all like unshared -> shared.

const only takes away capabilities, shared adds new capabilities, such as sending a reference to another thread. If you have two threads that share data, you need cooperation from both to properly synchronize accesses.

>>> Of course it can handle it threadsave, but as it is local,
>>> that is only overhead - reading or changing the value can't do
>>> any harm either. I like the idea.
>>>
>>>> But useless, because there is no way to ensure thread safety of reads
>>>> and writes if only one party to the shared state knows about the sharing.
>>> Of course there is.
>>
>> Please do enlighten me. You have two processors operating
>> (reading/writing) on the same address space on a modern computer
>> architecture with a weak memory model, and you are using an optimizing
>> compiler. How do you ensure sensible results without cooperation from
>> both of them? (Hint: you don't.)
> 
> What? This is a weird statement.
> So, you're saying that nobody has successfully written any threadsafe
> code, ever... we should stop trying, and we should admit that
> threadsafe queues and atomics, and mutexes and stuff all don't exist?
> 

Obviously I am not saying that.

>> without cooperation from both of them?
> 
> Perhaps this is the key to your statement?

Yes.

> Yes. 'cooperation from both of them' in this case means, they are both
> interacting with a threadsafe api, and they are blocked from accessing
> members, or any non-threadsafe api.
> ...

Yes. Your proposal only enforces this for the shared alias.

>>> Giving an unshared value to a function that
>>> even can handle shared values may create some overhead, but is
>>> indeed threadsave.
>>>
>>
>> Yes, if you give it to one function only, that is the case. However, as
>> you may know, concurrency means that there may be multiple functions
>> operating on the data _at the same time_. If one of them operates on the
>> data as if it was not shared, you will run into trouble.
> 
> Who's doing this,

Anyone, it really does not matter. One major point of the type system is to ensure that _all_ @safe code has defined behavior. You can convert between shared and unshared, just not in @safe code.

> and how?
> ...

They create a mutable instance of a class, they create a shared alias using one of your proposed holes, then send the shared alias to another thread, call some methods on it in both threads and get race conditions.

>> You are arguing as if there was either no concurrency or no mutable
>> aliasing.
> 
> If a class has no shared methods, there's no possibility for mutable aliasing.
> If the class has shared methods, then the class was carefully designed
> to be threadsafe.
> 

Not necessarily. Counterexample:

@safe:
class C{
    int x;
    void foo(){
        x+=1; // this can still race with atomicIncrement
    }
    void bar()shared{
        atomicIncrement(x); // presumably you want to allow this
    }
}

void main(){
    auto c=new C();
    shared s=c; // depending on your exact proposed rules, this step may be more cumbersome
    spawn!(()=>s.bar());
    s.foo(); // race
}

Now, if a class has only shared members, that is another story. In this case, all references should implicitly convert to shared. There's a DIP I meant to write about this. (For all qualifiers, not just shared).

On 16.10.2018 19:25, Manu wrote:
> On Tue, Oct 16, 2018 at 3:20 AM Timon Gehr via Digitalmars-d
> <digitalmars-d@puremagic.com> wrote:
>>
>> On 15.10.2018 20:46, Manu wrote:
>>>
>>> Assuming the rules above: "can't read or write to members", and the
>>> understanding that `shared` methods are expected to have threadsafe
>>> implementations (because that's the whole point), what are the risks
>>> from allowing T* -> shared(T)* conversion?
>>>
>>
>> Unshared becomes useless, and in turn, shared becomes useless. You can't
>> have unshared/shared aliasing.
> 
> What aliasing?

Aliasing means you have two references to the same data. The two references are then said to alias. An implicit conversion from unshared to shared by definition introduces aliasing, where one of the two references is unshared and the other is shared.

> Please show a reasonable and likely construction of the
> problem. I've been trying to think of it.
> ...

I have given you an example. I don't care whether it is "reasonable" or "likely". The point of @safe is to have a subset of the language with a sound type system.

>>> All the risks that I think have been identified previously assume that
>>> you can arbitrarily modify the data. That's insanity... assume we fix
>>> that... I think the promotion actually becomes safe now...?
>>
>> But useless, because there is no way to ensure thread safety of reads
>> and writes if only one party to the shared state knows about the sharing.
> 
> What? I don't understand this sentence.
> 
> If a shared method is not threadsafe, then it's an implementation
> error. A user should expect that a shared method is threadsafe,
> otherwise it shouldn't be a shared method! Thread-local (ie, normal)
> methods are for not-threadsafe functionality.
> 

Your function can be thread safe all you want. If you have a thread unsafe function also operating on the same state, you will still get race conditions. E.g. mutual exclusion is based on cooperation from all threads. If one of them forgets to lock, it does not matter how threadsafe all the others are.

On 15.10.2018 23:51, Manu wrote:
> If a shared method is incompatible with an unshared method, your class
> is broken.

Then what you want is not implicit unshared->shared conversion. What you want is a different way to type shared member access. You want a setup where shared methods are only allowed to access shared members and unshared methods are only allowed to access unshared members.

I.e., what you want is that shared is not transitive. You want that if you have a shared(C) c, then it is an error to access c.m iff m is not shared. This way you can have partially shared classes, where part of the class is thread-local, and other parts are shared with other threads.

Is this it?

On 17.10.2018 14:24, Timon Gehr wrote:
> On 15.10.2018 23:51, Manu wrote:
>> If a shared method is incompatible with an unshared method, your class
>> is broken.
> 
> Then what you want is not implicit unshared->shared conversion. What you want is a different way to type shared member access. You want a setup where shared methods are only allowed to access shared members and unshared methods are only allowed to access unshared members.
> 
> I.e., what you want is that shared is not transitive. You want that if you have a shared(C) c, then it is an error to access c.m iff m is not shared. This way you can have partially shared classes, where part of the class is thread-local, and other parts are shared with other threads.
> 
> Is this it?

(Also, with this new definition of 'shared', unshared -> shared conversion would of course become sound.)

On 17.10.2018 09:20, Manu wrote:
>> Timon Gehr has done a good job showing that they still stand
>> unbreached.
> His last comment was applied to a different proposal.
> His only comment on this thread wasn't in response to the proposal in
> this thread.
> If you nominate Timon as your proxy, then he needs to destroy my
> proposal, or at least comment on it, rather than make some prejudiced
> comment generally.
> 

There is no "prejudice", just reasoning. Your proposal was "disallow member access on shared aggregates, allow implicit conversion from unshared to shared and keep everything else the same". This is a bad proposal. There may be a good proposal that allows the things you want, but you have not stated what they are, your OP was just: "look at this bad proposal, it might work, no?" I said no, then was met with some hostility.

You should focus on finding a good proposal that achieves what you want without breaking the type system instead of attacking me.

On 17.10.2018 14:29, Timon Gehr wrote:
> to access c.m iff m is not shared

Unfortunate typo. This should be if, not iff (if and only if).

On 10/16/18 6:24 PM, Nicholas Wilson wrote:
> On Tuesday, 16 October 2018 at 21:19:26 UTC, Steven Schveighoffer wrote:
>> OK, so here is where I think I misunderstood your point. When you said a lock-free queue would be unusable if it wasn't shared, I thought you meant it would be unusable if we didn't allow the implicit cast. But I realize now, you meant you should be able to use a lock-free queue without it being actually shared anywhere.
>>
>> What I say to this is that it doesn't need to be usable. I don't care to use a lock-free queue in a thread-local capacity. I'll just use a normal queue, which is easy to implement, and doesn't have to worry about race conditions or using atomics. A lock free queue is a special thing, very difficult to get right, and only really necessary if you are going to share it. And used for performance reasons!
> 
> I think this comes up where the queue was originally shared, you acquired a lock on the thing it is a member of, and you want to continue using it through your exclusive reference.
> 

Isn't that a locking queue? I thought we were talking lock-free?

-Steve

On 17.10.2018 14:24, Timon Gehr wrote:
> and unshared methods are only allowed to access unshared members.

This is actually not necessary, let me reformulate:

You want:

- if you have a C c and a shared(C) s, typeof(s.x) == typeof(c.x).
- shared methods are not allowed to access unshared members.
- shared is not transitive, and therefore unshared class references implicitly convert to shared class references

Applied to pointers, this would mean that you can implicitly convert int* -> shared(int*), but not shared(int*)->int*, int* -> shared(int)* or shared(int)* -> int*. shared(int*) and shared(shared(int)*) would be different types, such that shared(int*) cannot be dereferenced but shared(shared(int)*) can.

On 10/16/18 8:26 PM, Manu wrote:
> On Tue, Oct 16, 2018 at 2:20 PM Steven Schveighoffer via Digitalmars-d
> <digitalmars-d@puremagic.com> wrote:
>>
>> On 10/16/18 4:26 PM, Manu wrote:
>>> On Tue, Oct 16, 2018 at 11:30 AM Steven Schveighoffer via
>>> Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>>> int x;
>>>>
>>>> shared int *p = &x; // allow implicit conversion, currently error
>>>>
>>>> passToOtherThread(p);
>>>>
>>>> useHeavily(&x);
>>>
>>> What does this mean? It can't do anything... that's the whole point here.
>>> I think I'm struggling here with people bringing presumptions to the
>>> thread. You need to assume the rules I define in the OP for the
>>> experiment to work.
>>
>> OK, I wrote a whole big response to this, and I went and re-quoted the
>> above, and now I think I understand what the point of your statement is.
>>
>> I'll first say that if you don't want to allow implicit casting of
>> shared to mutable,
> 
> It's critical that this is not allowed. It's totally unreasonable to
> cast from shared to thread-local without synchronisation.

OK, so even with synchronization in the second thread when you cast, you still have a thread-local pointer in the originating thread WITHOUT synchronization.

> It's as bad as casting away const.

Of course! But shared has a different problem from const. Const allows the data to change through another reference, shared cannot allow changes without synchronization.

Changes without synchronization are *easy* with an unshared reference. Data can't be shared and unshared at the same time.

>> then you can't allow implicit casting from mutable to
>> shared. Because it's mutable, races can happen.
> 
> I don't follow...

You seem to be saying that shared data is unusable. But why the hell have it then? At some point it has to be usable. And the agreed-upon use is totally defeated if you also have some stray non-shared reference to it.

> 
>> There is in fact, no difference between:
>>
>> int *p;
>> shared int *p2 = p;
>> int *p3 = cast(int*)p2;
> 
> Totally illegal!! You casted away shared. That's as bad as casting away const.

But if you can't do anything with shared data, how do you use it?

> 
>> and this:
>>
>> int *p;
>> shared int *p2 = p;
>> int *p3 = p;
> 
> There's nothing wrong with this... I don't understand the point?

It's identical to the top one. You now have a new unshared reference to shared data. This is done WITHOUT any agreed-upon synchronization.

>> So really, the effort to prevent the reverse cast is defeated by
>> allowing the implicit cast.
> 
> Only the caller has the thread-local instance. You can take a
> thread-local pointer to a thread-local within the context of a single
> thread.
> So, it's perfectly valid for `p` and `p3` to exist in a single scope.
> `p2` is fine here too... and if that shared pointer were to escape to
> another thread, it wouldn't be a threat, because it's not readable or
> writable, and you can't make it back into a thread-local pointer
> without carefully/deliberately deployed machinery.

Huh? If shared data can never be used, why have it?

Pretend that p is not a pointer to an int, but a pointer to an UNSHARED type that has shared methods on it and unshared methods (for when you don't need any sync).

Now the shared methods will obey the sync, but the unshared ones won't. The result is races. I can't understand how you don't see that.

>> There is a reason we disallow assigning from mutable to immutable
>> without a cast. Yet, it is done in many cases, because you are sometimes
>> building an immutable object with mutable pieces, and want to cast the
>> final result.
> 
> I don't think analogy to immutable has a place in this discussion, or
> at least, I don't understand the relevance...
> I think the reasonable analogy is const.

No, immutable is more akin to shared because immutable and mutable are completely different. const can point at mutable or immutable data. shared can't be both shared and unshared. There's no comparison. Data is either shared or not shared, there is no middle ground. There is no equivalent of const to say "this data could be shared, or could be unshared".

>> In this case, it's ON YOU to make sure it's correct, and the traditional
>> mechanism for the compiler giving you the responsibility is to require a
>> cast.
> 
> I think what you're talking about are behaviours relating to casting
> shared *away*, and that's some next-level shit. Handling in that case
> is no different to the way it exists today. You must guarantee that
> the pointer you possess becomes thread-local before casting it to a
> thread-local pointer.
> In my application framework, I will never cast shared away under my
> proposed design. We don't have any such global locks.

OK, so how does shared data actually operate? Somewhere, the magic has to turn into real code. If not casting away shared, what do you suggest?

>> -----
>>
>> OK, so here is where I think I misunderstood your point. When you said a
>> lock-free queue would be unusable if it wasn't shared, I thought you
>> meant it would be unusable if we didn't allow the implicit cast. But I
>> realize now, you meant you should be able to use a lock-free queue
>> without it being actually shared anywhere.
> 
> Right, a lock-free queue is a threadsafe object, and it's methods work
> whether the queue is shared or not.
> The methods are attributed shared because they can be called on shared
> instances... but they can ALSO be called from a thread-local instance,
> and under my suggested promotion rules, it's fine for the this-pointer
> to promote to shared to make the call.

It's fine in terms of a specific object we are talking about, with pre-determined agreements as to whether the data will be passed to other threads. But the compiler has no idea about these agreements. It can't logically determine that this is safe without you telling it, hence the requirement for casting.

>> What I say to this is that it doesn't need to be usable. I don't care to
>> use a lock-free queue in a thread-local capacity. I'll just use a normal
>> queue, which is easy to implement, and doesn't have to worry about race
>> conditions or using atomics. A lock free queue is a special thing, very
>> difficult to get right, and only really necessary if you are going to
>> share it. And used for performance reasons!
> 
> I'm more interested in the object that has that lock-free queue as a
> member... it is probably a mostly thread-local object, but may have a
> couple of shared methods.

I get that, I think the shared methods just need to have unshared versions for the case when the queue is fully thread-local.

> I have a whole lot of objects which have 3 tiers of API access; the
> thread-local part, the threadsafe part, and the const part. Just as a
> mutable instance can call a const method, there's no reason a
> thread-local instance can't call a threadsafe method.

If you call a const method, it can squirrel away a const pointer to the otherwise mutable data, which is then usable later (and safe to do so).

The same cannot be said for a shared pointer. That can be then easily moved to another thread, WITHOUT the expectation that it was. And in that case, the now thread-local queue is actually shared between threads, and calling the thread-local API will cause races.

I didn't respond to the rest of the comments, because they were simply another form of "shared is similar to const", which is not true.

threadsafe use of shared data depends on ALL threads using those same mechanisms. If one uses simple thread-local use, then it all falls apart.

-Steve

Jesus Manu, it's soon 8 pages of dancing around a trivial issue.

Implicit casting from mutable to shared is unsafe, case closed.
Explicit cast from mutable to unsafe, on the other hand:

- is an assertion (on programmer's behalf) that this instance is indeed unique
- is self-documenting
- is greppable (especially if implemented as assumeShared or other descriptive name).

I don't understand why are you so fixated on this. The other parts of your proposal are much more important.