On Thursday, 18 October 2018 at 17:17:37 UTC, Atila Neves wrote:
> [snip]
>
>> Assuming this world... how do you use shared?
>
> https://github.com/atilaneves/fearless
>

I had posted your library before to no response...

I had two questions, if you'll indulge me.

The first is perhaps more wrt automem. I noticed that I couldn't use automem's Unique with @safe currently. Is there any way to make it @safe, perhaps with dip1000?

Second, Rust's borrow checker is that you can only have one mutable borrow. This is kind of like Exclusive, but it does it at compile-time, rather than with GC/RC. Is this something that can be incorporated into fearless?

On Thursday, 18 October 2018 at 20:07:54 UTC, Stanislav Blinov wrote:
> On Thursday, 18 October 2018 at 19:51:17 UTC, Erik van Velzen wrote:
>> On Thursday, 18 October 2018 at 19:26:39 UTC, Stanislav Blinov
>
>>>> Manu said clearly that the receiving thread won't be able to read or write the pointer.
>>>
>>> Yes it will, by casting `shared` away. *Just like* his proposed "wrap everything into" struct will. There's exactly no difference.
>>>
>
>> Casting is inherently unsafe. Or at least, there's no threadsafe guarantee.
>
> So? That's the only way to implement required low-level access, especially if we imagine that the part of Manu's proposal about disabling reads and writes on `shared` values is a given. It's the only way to implement Manu's Atomic!int, or at least operation it requires, for example.
>
>>>> You can still disagree on the merits, but so far it has been demonstrated as a sound idea.
>>>
>>> No, it hasn't been.
>>
>> I think you are missing the wider point. I can write thread-unsafe code *right now*, no casts required. Just put shared at the declaration. The proposal would actually give some guarantees.
>
> No, I think you are missing the wider point. You can write thread-unsafe code regardless of using `shared`, and regardless of its implementation details. Allowing *implicit automatic promotion* of *mutable thread-local* data to shared will allow you to write even more thread-unsafe code, not less.

This may come as a surprise but I agree with this factually.

It's just that the extra guarantee provided by disallowing implicit casting to shared is not so valuable to me. If you have an object which can be used in both a thread-safe and a thread-unsafe way that's a bug or code smell. Like the earlier example with localInc() and sharedInc(). And we can't guarantee the safety of @trusted implementations anyways.

If the extra guarantee is not valuable, might as well allow it.

> The solid part of the proposal is about disabling reads and writes. The rest is pure convention: write structs instead of functions,

"Writing structs" seems acknowledging how it's usually done. And then you put these in a library with generic concurrent data structures.

> and somehow (???) benefit from a totally unsafe implicit cast.

(See first part)

Manu I'm also making a plea for you to write a document with your proposal which aggregates all relevant examples and objections. Then you can easily refer to it and we can introduce ppl to idea without reading a megathread.

On Thu, Oct 18, 2018 at 2:40 PM Erik van Velzen via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> Manu I'm also making a plea for you to write a document with your proposal which aggregates all relevant examples and objections. Then you can easily refer to it and we can introduce ppl to idea without reading a megathread.

I can start on this. I wouldn't take the time before having confidence
the whole effort was not to be rejected in principle.
I'm still not sure on this.
I think WRT Steven, we've understood each others position, and the
only point of contention is on weighting a value-judgement.
I'm confident in my weighting strategy, but I need to convince.

On Thursday, 18 October 2018 at 18:12:03 UTC, Stanislav Blinov wrote:
> On Thursday, 18 October 2018 at 18:05:51 UTC, aliak wrote:
>
>> Right, but the argument is a shared int*, so from what I've understood... you can't do anything with it since it has no shared members. i.e. you can't read or write to it. No?
>
> Obviously the implementation would cast `shared` away, just like it would if it were Atomic!int. But for some reason, Manu thinks that latter is OK doing that, but former is voodoo. Go figure.

Sounds like one is encapsulated within a box that carefully handles thread safety and makes promises with the API and the other is not.

I don't think you can apply shared on a free function, i.e.:

void increment(shared int*) shared;

in which case increment would not, and cannot be a threadsafe api in Manu's world.

So once you throw an Object in to shared land all you could do is call shared methods on it, and since they'd have been carefully written with sharing in mind... it does seem a lot more usable.

On these two cases:

increment(shared int* p1) {
 // I have no guarantees that protecting and accessing p1 will not cause problems
 //
 // but you don't have this guarantee in any world (current nor MP) because you can
 // never be sure that p1 was not cast from a mutable.
}

int* p2;
increment(p2);
// I have no guarantee that accessing p2 is safe anymore.
//
// But that would apply only if the author of increment was being unsafe.
// and "increment" cannot be marked as shared.

Manu, Erik, Simen... In what world can a person consciously say "casting is unsafe", and yet at the same time claim that *implicit casting* is safe? What the actual F, guys?

On Thursday, 18 October 2018 at 14:19:41 UTC, Steven Schveighoffer wrote:
> On 10/18/18 10:11 AM, Simen Kjærås wrote:
>> On Thursday, 18 October 2018 at 13:35:22 UTC, Steven Schveighoffer wrote:
>>> struct ThreadSafe
>>> {
>>>    private int x;
>>>    void increment()
>>>    {
>>>       ++x; // I know this is not shared, so no reason to use atomics
>>>    }
>>>    void increment() shared
>>>    {
>>>       atomicIncrement(&x); // use atomics, to avoid races
>>>    }
>>> }
>> 
>> But this isn't thread-safe, for the exact reasons described elsewhere in this thread (and in fact, incorrectly leveled at Manu's proposal). Someone could write this code:
>> 
>> void foo() {
>>      ThreadSafe* a = new ThreadSafe();
>>      shareAllOver(a);
>
> Error: cannot call function shareAllOver(shared(ThreadSafe) *) with type ThreadSafe *

Sorry, typo. Should of course have been shareAllOver(cast(shared)a);

>>      a.increment(); // unsafe, non-shared method call
>> }
>> 
>> When a.increment() is being called, you have no idea if anyone else is using the shared interface.
>
> I do, because unless you have cast the type to shared, I'm certain there is only thread-local aliasing to it.



>
>> This is one of the issues that MP (Manu's Proposal) tries to deal with. Under MP, your code would *not* be considered thread-safe, because the non-shared portion may interfere with the shared portion. You'd need to write two types:
>> 
>> struct ThreadSafe {
>>      private int x;
>>      void increment() shared {
>>          atomicIncrement(&x);
>>      }
>> }
>> 
>> struct NotThreadSafe {
>>      private int x;
>>      void increment() {
>>          ++x;
>>      }
>> }
>> 
>> These two are different types with different semantics, and forcing them both into the same struct is an abomination.
>
> Why? What if I wanted to have an object that is local for a while, but then I want it to be shared (and I ensure carefully when I cast to shared that there are no other aliases to that)?
>
>> In your case, the user of your type will need to ensure thread-safety.
>
> No, the contract the type provides is: if you DON'T cast unshared to shared or vice versa, the type is thread-safe.
>
> If you DO cast unshared to shared, then the type is thread-safe as long as you no longer use the unshared reference.
>
> This is EXACTLY how immutable works.

Yes, and that means the user of the type will need to follow these rules to ensure thread-safety. Which is what I said. Under MP, it's simply safe.


>> You may not have any control over how he's doing things, while you *do* control the code in your own type (and module, since that also affects things). Under MP, the type is what needs to be thread-safe, and once it is, the chance of a user mucking things up is much lower.
>
> Under MP, the type is DEFENSIVELY thread-safe, locking or using atomics unnecessarily when it's thread-local.

Yes, because safety >> efficiency. There's nothing stopping you from making fast, unsafe functions under MP. Call them unsafe_<foo> to make them easy to grep for. Since only the type implementer, not the users, write this code, it's not an undue burden.

One of the greatest benefits of MP is that all the potential problem points are in one place. In a large codebase with multiple developers, anyone anywhere could be using a dangling unshared reference under the current schema. Under MP, that's confined to the type, not its uses.

--
  Simen

On Thursday, 18 October 2018 at 21:54:55 UTC, Stanislav Blinov wrote:
> Manu, Erik, Simen... In what world can a person consciously say "casting is unsafe", and yet at the same time claim that *implicit casting* is safe? What the actual F, guys?

In a world where the implicit casting always ends in a place where anything you can do is safe. As we have said ad nauseam.

--
  Simen

On Thursday, 18 October 2018 at 16:31:02 UTC, Stanislav Blinov wrote:
> You contradict yourself and don't even notice it. Per your rules, the way to open that locked box is have shared methods that access data via casting. Also per your rules, there is absolutely no way for the programmer to control whether they're actually sharing the data. Therefore, some API can steal a shared reference without your approval, use that with your "safe" shared methods, while you're continuing to threat your data as not shared.

Yes, and that's fine. Because it's thread-safe, remember?


> You and Manu both seem to think that methods allow you to "define a thread-safe interface".
>
> struct S {
>     void foo() shared;
> }
>
> Per your rules, S.foo is thread-safe. It is here that I remind you, *again*, what S.foo actually looks like, given made-up easy-to-read mangling:
>
> void struct_S_foo(ref shared S);
>
> And yet, for some reason, you think that these are not thread-safe:
>
> void foo(shared int*);
> void bar(ref shared int);

Again, no. No. No, no, no, no, no. We have not said that, we are not saying that, and we will not say that. Because it's not true, and I pointed out exactly this in a previous post. I have no idea where you got this idea, and I hope we can excise it. There is absolutely nothing wrong with void foo(shared int*). (apart from the fact it can't safely do anything)

For clarity: the interface of a type is any method, function, delegate or otherwise that may affect its internals. That means any free function in the same module, and any non-private members.


> Your implementation of 'twaddle' is *unsafe*, because the compiler doesn't know that 'payload' is shared. For example, when inlining, it may reorder the calls in it and cause races or other UB. At least one of the reasons behind `shared` *was* to serve as compiler barrier.

Ah, now this is a good point - thanks! That does seem like it's a harder problem than has come up thus far.


>> Alright, so I have this shared object that I can't read from, and can't write to. It has no public shared members. What can I do with it? I can pass it to other guys, who also can't do anything with it. Are there other options?
>
> It can have any number of public shared "members" per UFCS. The fact that you forget is that there's no difference between a method and a free function, other than syntax sugar. Well, OK, there's guaranteed private access for methods, but same is true for module members.

This again? See point 2, above. I hope we can stop this silliness soon.


>>>>> The rest just follows naturally.
>>>
>>> Nothing follows naturally. The proposal doesn't talk at all about the fact that you can't have "methods" on primitives,
>>
>> You can't have thread-safe methods operating directly on primitives, because they already present a non-thread-safe interface. This is true. This follows naturally from the rules.
>
> Everything in D already presents a non-threadsafe interface. Things that you advocate included.
>
> struct S {
>     void foo() shared;
> }
>
> That is not threadsafe. This *sort of* is:
>
> struct S {
>     @disable this(this);
>     @disable void opAssign(S);
>
>     void foo() shared;
> }
>
> ...except not quite still. Now, if the compiler generated above in the presence of any `shared` members or methods, then we could begin talking about it being threadsafe. But that part is mysteriously missing from Manu's proposal, even though I keep reminding of this in what feels like every third post or so (I'm probably grossly exaggerating).

Again, this is good stuff. This is an actual example of what can go wrong. Thanks!


>>> that you can't distinguish between shared and unshared data if that proposal is realized,
>
>> And you can't do that currently either. Just like today, shared(T) means the T may or may not be shared with other thread. Nothing more, nothing less.
>
> I don't think it means what you think it means. "May or may not be shared with other thread" means "you MUST treat it as if it's shared with other thread". That's it.

Yup, hence 'shared' on a method meaning 'thread-safe'. So it's fine. I think this has been mentioned before.


> That's why automatic conversion doesn't make *any* sense, and that's why compiler error on attempting to pass over mutable as shared makes *perfect* sense.

No. Because shared access is thread-safe.


>>> that you absolutely destroy D's TLS-by-default treatment...
>> I'm unsure what you mean by this.
>
> You lose the ability to distinguish thread-local and shared data.

And when is this a problem? Again, anything that has shared access to something is incapable of doing anything non-thread-safe to it.


>>> Functions that you must not be allowed to write per this same proposal. How quaint.
>>
>> What? Which functions can't I write?
>
> Uh-huh, only due to some weird convention that "methods" are somehow safer than free functions. Which they're not.

No. Again, point 2. Nobody says this.


>> Yup, this is correct. But wrap it in a struct, like e.g. Atomic!int, and everything's hunky-dory.
>
> So again,
>
> void atomicInc(shared int*); // is "not safe", but
> void struct_Atomic_int_opUnary_plus_plus(ref shared Atomic); // is "safe"

No, void atomicInc(shared int*) is perfectly safe, as long as it doesn't cast away shared. Again, the problem is int already has a non-thread-safe interface, which Atomic!int doesn't. And once more, for clarity, this interface includes any function that has access to its private members, free function, method, delegate return value from a function/method, what have you. Since D's unit of encapsulation is the module, this has to be the case. For int, the members of that interface include all operators. For pointers, it includes deref and pointer arithmetic. For arrays indexing, slicing, access to .ptr, etc. None of these lists are necessarily complete.


>> I have no idea where I or Manu have said you can't make functions that take shared(T)*.
>
> Because you keep saying they're unsafe and that you should wrap them up in a struct for no other reason than just "because methods are kosher".

Again, point 2. I think we have been remiss in the explanation of what we consider the interface.


>> Let's say it together: for a type to be thread-safe, all of its public members must be written in a thread-safe way.
>
> It's shared private parts also must be written in a thread-safe way. Yes, they're private, but they still may be shared. Welcome to the communism of multithreading.

For the public members to be thread-safe, yes, the private parts must be thread-safe. That's always the case. If I'm going to build a skyscraper, I will not make the foundation out of cardboard.


Now, Two very good points came up in this post, and I think it's worth stating them again, because they do present possible issues with MP:

1) How does MP deal with reorderings in non-shared methods?

I don't know. I'd hide behind 'that's for the type implementor to handle', but it's a subtle enough problem that I'm not happy with that answer.


2) What about default members like opAssign and postblit?

The obvious solution is for the compiler to not generate these when a type has a shared method or is taken as shared by a free function in the same module. I don't like the latter part of that, but it should work.

--
  Simen

On Thu, Oct 18, 2018 at 2:55 PM Stanislav Blinov via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> Manu, Erik, Simen... In what world can a person consciously say "casting is unsafe", and yet at the same time claim that *implicit casting* is safe? What the actual F, guys?

Implicit casting exists in a world where the conversion is guaranteed
to be safe, because rules are defined appropriately.
Explicit casting exists in a world where those guarantees are not
present, because no such rules.

The 2 different strategies are 2 different worlds, one is my proposal,
the other is more like what we have now. They are 2 different
rule-sets.
You are super-attached to some presumptions, and appear to refuse to
analyse the proposal from the grounds it defines.