On Thursday, 18 October 2018 at 21:51:52 UTC, aliak wrote:
> On Thursday, 18 October 2018 at 18:12:03 UTC, Stanislav Blinov wrote:
>> On Thursday, 18 October 2018 at 18:05:51 UTC, aliak wrote:
>>
>>> Right, but the argument is a shared int*, so from what I've understood... you can't do anything with it since it has no shared members. i.e. you can't read or write to it. No?
>>
>> Obviously the implementation would cast `shared` away, just like it would if it were Atomic!int. But for some reason, Manu thinks that latter is OK doing that, but former is voodoo. Go figure.
>
> Sounds like one is encapsulated within a box that carefully

Unit of "encapsulation" in D is either a module or a package, not a struct. Free functions are a very valid way of accessing "encapsulated" data.

> handles thread safety and makes promises with the API and the other is not.

Nope.

void foo(const T* x);

makes a promise to not write through x. It assumes '*x' itself may not be const.

void foo(shared T* x);

makes a promise to threat '*x' in a thread-safe manner. But per MP, it *assumes* that '*x' is shared. And if it isn't, good luck finding that spot in your code.

> I don't think you can apply shared on a free function, i.e.:
>
> void increment(shared int*) shared;
>
> in which case increment would not, and cannot be a threadsafe api in Manu's world.

Wrong. In Manu's "world", this is somehow considered "safe":

void T_method_increment(ref shared T);

...because that is what a method is, while this:

void increment(shared T*);
void increment(ref shared T);

...is considered "unsafe" because reasons. Do you see the difference in signatures? I sure don't.

> So once you throw an Object in to shared land all you could do is call shared methods on it, and since they'd have been carefully written with sharing in mind... it does seem a lot more usable.

Same goes with free functions.

> On these two cases:
>
> increment(shared int* p1) {
>  // I have no guarantees that protecting and accessing p1 will not cause problems
>  //
>  // but you don't have this guarantee in any world (current nor MP) because you can
>  // never be sure that p1 was not cast from a mutable.
> }

Except that you *have to* *explicitly* cast it, which is:
a) documentation
b) greppable
c) easily fails review for people not authorized to do so

> int* p2;
> increment(p2);
> // I have no guarantee that accessing p2 is safe anymore.
> // But that would apply only if the author of increment was being unsafe.
> // and "increment" cannot be marked as shared.

No. *You*, the caller of an API (the "increment"), do not necessarily control that API. By allowing implicit conversion you waive all claims on your own data. In Manu's world, "increment" *assumes* you're doing the right thing. Yet at the same time, Manu happily talks about how only "experts" can do the right thing. How these two things co-exist in his world, I have no idea.

The "have no guarantee" holds in both cases. Except case (1) would require actually checking what the hell you're doing before making a cast, while in case (2) you just blindly write unsafe code.

On 10/18/18 5:22 PM, Manu wrote:
> On Thu, Oct 18, 2018 at 12:15 PM Steven Schveighoffer via
> Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>
>> On 10/18/18 2:55 PM, Manu wrote:
>>> On Thu, Oct 18, 2018 at 7:20 AM Steven Schveighoffer via Digitalmars-d
>>> <digitalmars-d@puremagic.com> wrote:
>>>>
>>>> On 10/18/18 10:11 AM, Simen Kjærås wrote:
>>>>> On Thursday, 18 October 2018 at 13:35:22 UTC, Steven Schveighoffer wrote:
>>>>>> struct ThreadSafe
>>>>>> {
>>>>>>      private int x;
>>>>>>      void increment()
>>>>>>      {
>>>>>>         ++x; // I know this is not shared, so no reason to use atomics
>>>>>>      }
>>>>>>      void increment() shared
>>>>>>      {
>>>>>>         atomicIncrement(&x); // use atomics, to avoid races
>>>>>>      }
>>>>>> }
>>>>>
>>>>> But this isn't thread-safe, for the exact reasons described elsewhere in
>>>>> this thread (and in fact, incorrectly leveled at Manu's proposal).
>>>>> Someone could write this code:
>>>>>
>>>>> void foo() {
>>>>>        ThreadSafe* a = new ThreadSafe();
>>>>>        shareAllOver(a);
>>>>
>>>> Error: cannot call function shareAllOver(shared(ThreadSafe) *) with type
>>>> ThreadSafe *
>>>
>>> And here you expect a user to perform an unsafe-cast (which they may
>>> not understand), and we have no language semantics to enforce the
>>> transfer of ownership. How do you assure that the user yields the
>>> thread-local instance?
>>
>> No, I expect them to do:
>>
>> auto a = new shared(ThreadSafe)();
> 
> I don't have any use for this design in my application.
> I can't use the model you prescribe, at all.

Huh? This is the same thing you are asking for. How were you intending to make a thread-safe thing sharable? Surely it will be typed as shared, right? How else will you pass it to multiple threads?

> 
>>> I think requiring the cast is un-principled in every way that D values.
>>
>> No cast is required. If you have shared data, it's shared. If you have
>> thread local data, it's unshared. Allocate the data the way you expect
>> to use it.
> 
> All data is thread-local, and occasionally becomes shared during periods.
> I can't make use of the model you describe.

If data is shared, it is shared. Once it is shared, it never goes back.

In your model, everything is *assumed* shared, so that's what you need to do, initialize it as shared. It still works just as you like. Even if you never actually share it, or share it periodically.

> My proposal is more permissive, and allows a wider range of
> application designs. What are the disadvantages?

The opposite is true. More designs are allowed by restricting casting as I have demonstrated many times.

>> It's only if you intend to turn unshared data into shared data where you
>> need an unsafe cast.
> 
> It's unnecessary though, because threadsafe functions are threadsafe!
> You're pointlessly forcing un-safety. Why would I prefer a design that
> forces unsafe interactions to perform safe operations?

No unsafe interactions are required for a type that defensively is shared. Just make it always shared, and you don't have any problems.

>> It's not even as difficult as immutable, because you can still modify
>> shared data. For instance, the shared constructor doesn't have to have
>> special rules about initialization, it can just assume shared from the
>> beginning.
> 
> Your design us immutable, mine is const.

No, your design is not const, const works on normal types. It's applicable to anything.

Your design is only applicable to special types that experts write. It's not applicable to int, for instance. It feels more like a special library than a compiler feature.

> Tell me, how many occurrences of 'immutable' can you find in your
> software? ... how about const?

I generally use inout whenever possible, or const when that is more appropriate. But that is for methods.

For data, I generally use immutable when I want a constant.

But like I said, something can't be both shared and unshared. So having shared pointers point at unshared data makes no sense -- once it's shared, it's shared. So shared really can't be akin to const.

> Which is more universally useful? If you had to choose one or the
> other, which one could you live without?

I would hate to have a const where you couldn't read the data, I probably would rather have immutable.

I said I would stop commenting on this thread, and I didn't keep that promise. I really am going to stop now. I'm pretty sure Walter will not agree with this mechanism, so until you convince him, I don't really need to be spending time on this.

We seem to be completely understanding each others mechanisms, but not agreeing which one is correct, based on (from both sides) hypothetical types and usages.

-Steve

On Thursday, 18 October 2018 at 22:08:14 UTC, Simen Kjærås wrote:
> On Thursday, 18 October 2018 at 16:31:02 UTC, Stanislav Blinov

>> Now, if the compiler generated above in the presence of any `shared` members or methods, then we could begin talking about it being threadsafe...

> Again, this is good stuff. This is an actual example of what can go wrong. Thanks!

You're welcome.

> No, void atomicInc(shared int*) is perfectly safe, as long as it doesn't cast away shared.

Eh? If you can't read or write to your shared members, how *do* you implement your "safe" shared methods without casting away shared? Magic?!

> Again, the problem is int already has a non-thread-safe interface, which Atomic!int doesn't.

*All* structs and classes have a non-thread-safe interface, as I have demonstrated, and thankfully you agree with that. But *there is no mention of it* in the OP, nor there was *any recognition* of it up to this point. When I asked about copying, assignment and destructors in the previous thread (thread, not post), Manu quite happily proclaimed those were fine given some arbitrary conditions. Yet those are quite obviously *not fine*, especially *if* you want to have a "safe" implicit conversion. *That* prompted me to assume that Manu didn't actually think long and hard about his proposal and what it implies. Without recognizing those issues:

struct S {
    private int x;
    void foo() shared;
}

void shareWithThread(shared S* s);

auto s = make!S;     // or new, whatever
shareWithThread(&s); // Manu's implicit conversion

// 10 kLOC below, written by some other guy 2 years later:
*s = S.init;

^ that is a *terrible*, and un-greppable BUG. How fast would you spot that in a review? Pretty fast if you saw or wrote the other code yesterday. A week later? A month? A year?..

Again, that is assuming *only* what I'm *certain about* in Manu's proposal, not something he or you assumed but didn't mention.

> And once more, for clarity, this interface includes any function that has access to its private members, free function, method, delegate return value from a function/method, what have you. Since D's unit of encapsulation is the module, this has to be the case. For int, the members of that interface include all operators. For pointers, it includes deref and pointer arithmetic. For arrays indexing, slicing, access to .ptr, etc. None of these lists are necessarily complete.

Aight, now, *now* I can perhaps try to reason about this from your point of view. Still, I would need some experimentation to see if such approach could actually work. And that would mean digging out old non-`shared`-aware code and performing some... dubious... activities.

> I have no idea where I or Manu have said you can't make functions that take shared(T)*.

Because that was the only way to reason about your interpretations of various examples until you said this:

> I think we have been remiss in the explanation of what we consider the interface.
> For clarity: the interface of a type is any method, function, delegate or otherwise that may affect its internals. That means any free function in the same module, and any non-private members.

Now compare that to what is stated in the OP and correlate with what I'm saying, you might understand where my opposition comes from.

> Now, Two very good points came up in this post, and I think it's worth stating them again, because they do present possible issues with MP:
>
> 1) How does MP deal with reorderings in non-shared methods?
>
> I don't know. I'd hide behind 'that's for the type implementor to handle', but it's a subtle enough problem that I'm not happy with that answer.
>
>
> 2) What about default members like opAssign and postblit?
>
> The obvious solution is for the compiler to not generate these when a type has a shared method or is taken as shared by a free function in the same module. I don't like the latter part of that, but it should work.

Something I didn't yet stress about (I think only mentioned briefly somewhere) is, sigh, destructors. Right now, `shared` allows you to either have a `~this()` or a `~this() shared`, but not both. In my mind, `~this() shared` is an abomination. One should either:

1) have data that starts life shared (a global, or e.g. new shared(T)), and simply MUST NOT have a destructor. Such data is ownerless, or you can say that everybody owns it. Therefore there's no deterministic way of knowing whether or not or when to call the destructor. You can think of it as an analogy with current stance on finalizers with GC.

2) have data that starts life locally (e.g. it's not declared `shared`, but converted later). Such types MAY have a destructor, because they always have a cleanly defined owner: whoever holds the non-`shared` reference (recall that copying MUST be *disabled* for any shared-aware type). But that destructor MUST NOT be `shared`.

Consequently, types such as these:

shared struct S { /* ... */ }

MUST NOT define a destructor, either explicitly, or implicitly through members, i.e. it's a compile error if an __xdtor needs to be generated for such type.

However, in practice this would mean that practically all types couldn't have a destructor:

struct S {
    private shared X x; // X must not have a destructor, even though S can?..
}

Perhaps, an exception to (1) above could be made for such cases, but I'm too tired to think about wording at the moment.

Also, the proposal has no mention of interaction of `shared` and the GC, which AFAIK is also missing pretty much everywhere you can even get some information on current state of `shared`. This *needs* to be addressed.

On Thursday, 18 October 2018 at 22:09:02 UTC, Manu wrote:

> The 2 different strategies are 2 different worlds, one is my proposal,
> the other is more like what we have now. They are 2 different
> rule-sets.
> You are super-attached to some presumptions, and appear to refuse to
> analyse the proposal from the grounds it defines.

Please see my exchange with Simen in case you're skipping my posts.

On 18.10.18 20:26, Steven Schveighoffer wrote:
>>
>> i = 1;
>> int x = i;
>> shared int y = i;
> 
> This should be fine, y is not shared when being created.
> 
> However, this still is allowed, and shouldn't be:
> 
> y = 5;
> 
> -Steve

I'm pretty sure you will have to allow operations on shared local variables. Otherwise, how are you ever going to use a shared(C)? You can't even call a shared method on it because it involves reading the reference.

On 18.10.18 23:34, Erik van Velzen wrote:
> If you have an object which can be used in both a thread-safe and a thread-unsafe way that's a bug or code smell.

Then why do you not just make all members shared? Because with Manu's proposal, as soon as you have a shared method, all members effectively become shared. It just seems pointless to type them as unshared anyway and then rely on convention within @safe code to prevent unsafe accesses. Because, why? It just makes no sense.

With the proposal I posted in the beginning, you would then not only get implicit conversion of class references to shared, but also back to unshared.

I think the conflation of shared member functions and thread safe member functions is confusing. shared on a member function just means that the `this` reference is shared. The only use case for this is overloading on shared. The D approach to multithreading is that /all/ functions should be thread safe, but it is easier for some of them because they don't even need to access any shared state. It is therefore helpful if the type system cleanly separates shared from unshared state.

On Thursday, 18 October 2018 at 23:47:56 UTC, Timon Gehr wrote:

> I'm pretty sure you will have to allow operations on shared local variables. Otherwise, how are you ever going to use a shared(C)? You can't even call a shared method on it because it involves reading the reference.

Because you can't really "share" C (e.g. by value). You share a C*, or, rather a shared(C)*. The pointer itself, which you own, isn't shared at all, and shouldn't be: it's your own reference to shared data. You can read and write that pointer all you want. What you must not be able to do is read and write *c.

Although, when it's a global?.. I'm not sure. We can have the compiler always generate a by-reference access, i.e. make that part of the language spec. Because full-on read of C.sizeof can't be statically proven thread-safe generically anyway (that's why generated copying and assignment don't make any sense for `shared`).

On 19.10.18 02:29, Stanislav Blinov wrote:
> On Thursday, 18 October 2018 at 23:47:56 UTC, Timon Gehr wrote:
> 
>> I'm pretty sure you will have to allow operations on shared local variables. Otherwise, how are you ever going to use a shared(C)? You can't even call a shared method on it because it involves reading the reference.
> 
> Because you can't really "share" C (e.g. by value). You share a C*, or, rather a shared(C)*.

(Here, I intended C to be a class, if that was unclear.)

> The pointer itself, which you own, isn't shared at all, and shouldn't be: it's your own reference to shared data. You can read and write that pointer all you want. What you must not be able to do is read and write *c.
> ...

Presumably you could have a local variable shared(C) c, then take its address &c and send it to a thread which will be terminated before the scope of the local variable ends.

So, basically, the lack of tail-shared is an issue.

On Friday, 19 October 2018 at 00:36:11 UTC, Timon Gehr wrote:
> On 19.10.18 02:29, Stanislav Blinov wrote:
>> On Thursday, 18 October 2018 at 23:47:56 UTC, Timon Gehr wrote:
>> 
>>> I'm pretty sure you will have to allow operations on shared local variables. Otherwise, how are you ever going to use a shared(C)? You can't even call a shared method on it because it involves reading the reference.
>> 
>> Because you can't really "share" C (e.g. by value). You share a C*, or, rather a shared(C)*.
>
> (Here, I intended C to be a class, if that was unclear.)

In that case, it's already a pointer, and the only real issue is the interaction with GC, which I mentioned before *needs* to be addressed. And that is only when C was allocated by GC.

>> The pointer itself, which you own, isn't shared at all, and shouldn't be: it's your own reference to shared data. You can read and write that pointer all you want. What you must not be able to do is read and write *c.
>> ...
>
> Presumably you could have a local variable shared(C) c, then take its address &c and send it to a thread which will be terminated before the scope of the local variable ends.

I assume you mean *after*, because if that thread terminates before there's no problem.

> So, basically, the lack of tail-shared is an issue.

Well, not exactly. Irrespective of Manu's proposal, it's just inherent in D: sharing implies escaping, there's really no way around it. Provisions must be made in DIP1000 and in the language in general. However, that is a good point *against* implicit conversions, let alone @safe ones.

On Thu, Oct 18, 2018 at 3:40 PM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 10/18/18 5:22 PM, Manu wrote:
> > On Thu, Oct 18, 2018 at 12:15 PM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>
> >> On 10/18/18 2:55 PM, Manu wrote:
> >>> On Thu, Oct 18, 2018 at 7:20 AM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>>>
> >>>> On 10/18/18 10:11 AM, Simen Kjærås wrote:
> >>>>> On Thursday, 18 October 2018 at 13:35:22 UTC, Steven Schveighoffer wrote:
> >>>>>> struct ThreadSafe
> >>>>>> {
> >>>>>>      private int x;
> >>>>>>      void increment()
> >>>>>>      {
> >>>>>>         ++x; // I know this is not shared, so no reason to use atomics
> >>>>>>      }
> >>>>>>      void increment() shared
> >>>>>>      {
> >>>>>>         atomicIncrement(&x); // use atomics, to avoid races
> >>>>>>      }
> >>>>>> }
> >>>>>
> >>>>> But this isn't thread-safe, for the exact reasons described elsewhere in this thread (and in fact, incorrectly leveled at Manu's proposal). Someone could write this code:
> >>>>>
> >>>>> void foo() {
> >>>>>        ThreadSafe* a = new ThreadSafe();
> >>>>>        shareAllOver(a);
> >>>>
> >>>> Error: cannot call function shareAllOver(shared(ThreadSafe) *) with type
> >>>> ThreadSafe *
> >>>
> >>> And here you expect a user to perform an unsafe-cast (which they may not understand), and we have no language semantics to enforce the transfer of ownership. How do you assure that the user yields the thread-local instance?
> >>
> >> No, I expect them to do:
> >>
> >> auto a = new shared(ThreadSafe)();
> >
> > I don't have any use for this design in my application.
> > I can't use the model you prescribe, at all.
>
> Huh? This is the same thing you are asking for. How were you intending to make a thread-safe thing sharable? Surely it will be typed as shared, right? How else will you pass it to multiple threads?

Things get promoted to shared and distributed on occasion, but there is only one owner, and he's the only guy with thread-local access.

> >>> I think requiring the cast is un-principled in every way that D values.
> >>
> >> No cast is required. If you have shared data, it's shared. If you have thread local data, it's unshared. Allocate the data the way you expect to use it.
> >
> > All data is thread-local, and occasionally becomes shared during periods. I can't make use of the model you describe.
>
> If data is shared, it is shared. Once it is shared, it never goes back.

I'm suggesting that; data *may be shared* (ie, has valid threadsafe
interaction), or it may not (has no access).
NOT that data "is shared", or "is not". I don't see value in that
proposition. I don't know how to make that useful to me. It's
impossible for me to interact with that model safely.
The ONLY way to interact with that model safely is to allocate objects
shared, or not... there is no safe transition, and I am totally
concerned with the transition.

> In your model, everything is *assumed* shared, so that's what you need to do, initialize it as shared. It still works just as you like. Even if you never actually share it, or share it periodically.

You can't do that... cus you can't call anything's unshared methods.

> > My proposal is more permissive, and allows a wider range of application designs. What are the disadvantages?
>
> The opposite is true. More designs are allowed by restricting casting as I have demonstrated many times.

No you haven't.
What design is enabled by it that is inhibited by my design? I can't see this...
Your demonstrated design if fundamentally unsafe, unless you
*allocate* things as shared, and accept that no transition is
possible.
This is not a flexible design. My design also allows that if that's
what you want. Under my proposal, you can equally allocate things
shared to the exact same effect; I haven't taken anything away.

> >> It's only if you intend to turn unshared data into shared data where you need an unsafe cast.
> >
> > It's unnecessary though, because threadsafe functions are threadsafe! You're pointlessly forcing un-safety. Why would I prefer a design that forces unsafe interactions to perform safe operations?
>
> No unsafe interactions are required for a type that defensively is shared. Just make it always shared, and you don't have any problems.

Then you completely lose access to the unshared API. This is not workable.

> >> It's not even as difficult as immutable, because you can still modify shared data. For instance, the shared constructor doesn't have to have special rules about initialization, it can just assume shared from the beginning.
> >
> > Your design us immutable, mine is const.
>
> No, your design is not const, const works on normal types. It's applicable to anything.

But... that's the same here.
const restricts you to calling const methods... shared restricts you
to calling shared methods.

> > Tell me, how many occurrences of 'immutable' can you find in your software? ... how about const?
>
> I generally use inout whenever possible, or const when that is more appropriate. But that is for methods.
>
> For data, I generally use immutable when I want a constant.

You're dodging the question... I think you know that nothing can work
without const, but we could live without immutable.
Certainly, immutable wouldn't work without const!
The exact same connundrum applies here. Imagine that we only had
immutable, and any time you wanted to call a const method, you had to
cast to immutable...

> But like I said, something can't be both shared and unshared. So having shared pointers point at unshared data makes no sense -- once it's shared, it's shared. So shared really can't be akin to const.

Yes, but I'm saying the mutually-exclusive state isn't useful. No
transition is possible, and necessitates unsafety to do anything (that
I want to do).
I'm trying to create a world where the mutual-exclusion is effected in
practise (can define and perform threadsafe interaction), but not in
strict terms, such that transition is impossible... I'm very carefully
designing a solution where the worlds aren't isolated, and that world
is 100% more useful to me and my entire ecosystem.
I don't know how to use shared safely if it's not designed that way,
just like I don't know how to generally use immutable in lieu of
const. immutable wouldn't be practical without const.

> > Which is more universally useful? If you had to choose one or the other, which one could you live without?
>
> I would hate to have a const where you couldn't read the data, I probably would rather have immutable.

If an object has no const methods, you can't do anything with it. The usefulness of const depends on people recognising and support const.

> I said I would stop commenting on this thread, and I didn't keep that promise. I really am going to stop now. I'm pretty sure Walter will not agree with this mechanism, so until you convince him, I don't really need to be spending time on this.
>
> We seem to be completely understanding each others mechanisms, but not agreeing which one is correct, based on (from both sides) hypothetical types and usages.

I'm still not at all clear on how I'm excluding any interesting use
cases... I have deliberately tried to preserve all the valuable use
cases I'm aware of.
I don't understand your perspective, because I don't understand your
sense of loss.