On Tue, Aug 4, 2020 at 11:55 PM Sebastiaan Koppe via Digitalmars-d < digitalmars-d@puremagic.com> wrote:

> On Tuesday, 4 August 2020 at 12:52:01 UTC, Manu wrote:
> > On Tue, Aug 4, 2020 at 5:40 PM Sebastiaan Koppe via Digitalmars-d < digitalmars-d@puremagic.com> wrote:
> >> There is just one thing about shared I don't understand. If I design my object such that the non-shared methods are to be used thread-local and the shared methods from any thread, it follows that I should be able to call said shared methods from both a shared and non-shared instance of that object.
> >
> > Yes, this is a thing I talked at great length 1-2 years ago.
> > If you take shared to mean "is thread-safe", then my idea was
> > that
> > not-shared -> shared implicit conversion should be possible.
> > What I often do is this:
> >
> > struct Thing
> > {
> >   ref shared(Thing) implSharedCast() { return
> > *cast(shared)&this; }
> >   alias implSharedCast this;
> > }
> >
> > If that were an implicit conversion, that implies a slight
> > change of
> > meaning of shared (to one that I consider immensely more
> > useful), but it's
> > more challenging for the compiler to prove with confidence, and
> > there's a
> > lot of resistance to this change.
>
> What exactly does the compiler need to prove?


It would be ideal if the compiler could validate that the un-shared methods
are safe with respect to any shared methods.
If a shared (ie, 'thread-safe') method were called in parallel to an
un-shared method as is possible if the implicit conversion was allowed,
then it is necessary that un-shared methods need to be tolerant of their
'thread-safe' API being called at any time. That's the change in meaning
which is highly dangerous without some serious tooling. Essentially, adding
a shared method to an object applies a contract to all other NOT-shared
methods, and that doesn't feel good; "touching this here affects those
other things"... it will be super hard to audit.

Technology that looks like ThreadSanitizer might help, but it gets more tricky again when UFCS is used instead of member functions; those associations are harder to track.

I think that idea is doomed, and more recently I'm thinking about ways to prevent sneaking un-shared aliases across the conversion boundary... but I don't know how to achieve that without some sort of aliasing rules which seem unlikely.

The restrictions
> are all in place, you can only call shared methods on a shared object, and you can only access shared members in a shared method.
>

Ummmm no, under any of my proposals, you can't access shared members in a shared method... a shared method's `this` pointer is shared, which means all members are shared, and shared data may not be read or written at all... shared methods have no @safe access to members. They can only call other shared methods @safe-ly.

Hmm, well I guess if members themselves were also implicitly
> promoted to shared that could cause havoc.
>

Members inherit the qualifier of methods... in a shared method, members are shared, which means you can't read or write them.

On Wed, Aug 5, 2020 at 3:40 AM Timon Gehr via Digitalmars-d < digitalmars-d@puremagic.com> wrote:

> On 04.08.20 15:52, Sebastiaan Koppe wrote:
> > On Tuesday, 4 August 2020 at 12:52:01 UTC, Manu wrote:
> >> On Tue, Aug 4, 2020 at 5:40 PM Sebastiaan Koppe via Digitalmars-d < digitalmars-d@puremagic.com> wrote:
> >>> There is just one thing about shared I don't understand. If I design my object such that the non-shared methods are to be used thread-local and the shared methods from any thread, it follows that I should be able to call said shared methods from both a shared and non-shared instance of that object.
> >>
> >> Yes, this is a thing I talked at great length 1-2 years ago.
> >> If you take shared to mean "is thread-safe", then my idea was that
> >> not-shared -> shared implicit conversion should be possible.
> >> What I often do is this:
> >>
> >> struct Thing
> >> {
> >>   ref shared(Thing) implSharedCast() { return *cast(shared)&this; }
> >>   alias implSharedCast this;
> >> }
> >>
> >> If that were an implicit conversion, that implies a slight change of
> >> meaning of shared (to one that I consider immensely more useful), but
> >> it's
> >> more challenging for the compiler to prove with confidence, and there's
> a
> >> lot of resistance to this change.
> >
> > What exactly does the compiler need to prove? The restrictions are all
> > in place, you can only call shared methods on a shared object, and you
> > can only access shared members in a shared method.
> > ...
>
> Nope. You can access all members, but they will be treated as `shared`.
>

"treated as `shared`"; which is, under the `-preview` merged last year some
time, that they can not be read or written at all.
So you can take the address of all members, but not read or write to them.

> Hmm, well I guess if members themselves were also implicitly promoted to
> > shared that could cause havoc.
>
> Which is what the code above does, and also what Manu's "thread safe" qualifier would do. `shared` would no longer mean "shared".
>

I described an alternative approach in my reply to you. If there is any way to inhibit an un-shared alias from the caller being propagated to the callee, maintaining a separation at the call boundary between the unshared and shared instances, then it's safe without any of my other speculative rules.

On Sunday, August 2, 2020 7:23:26 PM MDT Andrei Alexandrescu via Digitalmars-d wrote:
> On 8/2/20 6:55 PM, Stefan Koch wrote:
> > shared means simply: this is not thread local,
> > therefore this thing might be used by other threads
> > therefore you read or write it in an expression directly.
> > You can however have a function take a shared argument and leave
> > the scheduling or locking or whatever synchronization you use,
> > to enact an operation in a safe way.
>
> Yes, the intent was generous. The realization, not quite. However, I did learn from Walter that a number of invalid operations on shared numerics have now been disabled. However, copying a shared int into an int is allowed and as far as I know with no special treatment:
>
> https://run.dlang.io/is/ZP124F
>
> At least on ARM that should generate a read barrier.

In princip@le, shared should be preventing any and all read-write operations which it cannot guarantee are atomic (which probably means banning all read-write operations, since that's going to vary across architectures). All code with shared then either needs to either use atomics or cast away shared and use mutexes, semaphores, or whatever other threading primitive is appropriate to ensure that the data is only accessed by that single thread while it's typed as thread-local, leaving it up to the programmer to then ensure that they're dealing with the threading correctly (just like in C++). However, unlike C++, all of the code operating on data shared across threads would then be segregated (it would also likely be @system in most cases). More complex objects can then have shared member functions which deal with all of the threading stuff internally rather than requiring that everyone using them deal with it directly, but at the core, it's ultimately going to be a question of atomics or casts if we want the guarantee that shared objects aren't being accessed across threads when they're not properly protected.

However, the core problem with all of this is that the restrictions that need to be put on shared have never really been put in place. For years, about all that it really did was disallow implicit conversions between shared and thread-local (and even then, it didn't always do it - e.g. with integers). Over time, additional restrictions have been added (I think mostly to numeric types), but it's all piecemeal. IIRC, there was a DIP that was supposed to outright make reading and writing to shared variables illegal, but I'm not quite sure where that stands. We also potentially need some language improvements to make it cleaner, because otherwise in order to read or write value types, we'll have to do stuff like take their address and cast the resulting pointer so that we can operate on the object as thread-local while the mutex is locked (or whatever or threading protections are in place).

So, while I think that the basic idea of shared is solid, it's never been properly implemented.

- Jonathan M Davis

On Thursday, August 13, 2020 12:44:54 AM MDT Jonathan M Davis via Digitalmars- d wrote:
> On Sunday, August 2, 2020 7:23:26 PM MDT Andrei Alexandrescu via
> Digitalmars-d
> wrote:
> > On 8/2/20 6:55 PM, Stefan Koch wrote:
> > > shared means simply: this is not thread local,
> > > therefore this thing might be used by other threads
> > > therefore you read or write it in an expression directly.
> > > You can however have a function take a shared argument and leave
> > > the scheduling or locking or whatever synchronization you use,
> > > to enact an operation in a safe way.
> >
> > Yes, the intent was generous. The realization, not quite. However, I did learn from Walter that a number of invalid operations on shared numerics have now been disabled. However, copying a shared int into an int is allowed and as far as I know with no special treatment:
> >
> > https://run.dlang.io/is/ZP124F
> >
> > At least on ARM that should generate a read barrier.
>
> In princip@le, shared should be preventing any and all read-write operations which it cannot guarantee are atomic (which probably means banning all read-write operations, since that's going to vary across architectures). All code with shared then either needs to either use atomics or cast away shared and use mutexes, semaphores, or whatever other threading primitive is appropriate to ensure that the data is only accessed by that single thread while it's typed as thread-local, leaving it up to the programmer to then ensure that they're dealing with the threading correctly (just like in C++). However, unlike C++, all of the code operating on data shared across threads would then be segregated (it would also likely be @system in most cases). More complex objects can then have shared member functions which deal with all of the threading stuff internally rather than requiring that everyone using them deal with it directly, but at the core, it's ultimately going to be a question of atomics or casts if we want the guarantee that shared objects aren't being accessed across threads when they're not properly protected.
>
> However, the core problem with all of this is that the restrictions that need to be put on shared have never really been put in place. For years, about all that it really did was disallow implicit conversions between shared and thread-local (and even then, it didn't always do it - e.g. with integers). Over time, additional restrictions have been added (I think mostly to numeric types), but it's all piecemeal. IIRC, there was a DIP that was supposed to outright make reading and writing to shared variables illegal, but I'm not quite sure where that stands. We also potentially need some language improvements to make it cleaner, because otherwise in order to read or write value types, we'll have to do stuff like take their address and cast the resulting pointer so that we can operate on the object as thread-local while the mutex is locked (or whatever or threading protections are in place).
>
> So, while I think that the basic idea of shared is solid, it's never been properly implemented.

LOL. And reading through the rest of the thread, I see that Manu already brought this up and that the DIP has an implementation with the -preview flag.

- Jonathan M Davis

On Thu, Aug 13, 2020 at 5:24 PM Jonathan M Davis via Digitalmars-d < digitalmars-d@puremagic.com> wrote:

> On Thursday, August 13, 2020 12:44:54 AM MDT Jonathan M Davis via
> Digitalmars-
> d wrote:
> > On Sunday, August 2, 2020 7:23:26 PM MDT Andrei Alexandrescu via
> > Digitalmars-d
> > wrote:
> > > On 8/2/20 6:55 PM, Stefan Koch wrote:
> > > > shared means simply: this is not thread local,
> > > > therefore this thing might be used by other threads
> > > > therefore you read or write it in an expression directly.
> > > > You can however have a function take a shared argument and leave
> > > > the scheduling or locking or whatever synchronization you use,
> > > > to enact an operation in a safe way.
> > >
> > > Yes, the intent was generous. The realization, not quite. However, I
> did
> > > learn from Walter that a number of invalid operations on shared
> numerics
> > > have now been disabled. However, copying a shared int into an int is allowed and as far as I know with no special treatment:
> > >
> > > https://run.dlang.io/is/ZP124F
> > >
> > > At least on ARM that should generate a read barrier.
> >
> > In princip@le, shared should be preventing any and all read-write
> operations
> > which it cannot guarantee are atomic (which probably means banning all read-write operations, since that's going to vary across architectures). All code with shared then either needs to either use atomics or cast away shared and use mutexes, semaphores, or whatever other threading primitive is appropriate to ensure that the data is only accessed by that single thread while it's typed as thread-local, leaving it up to the programmer
> to
> > then ensure that they're dealing with the threading correctly (just like
> in
> > C++). However, unlike C++, all of the code operating on data shared
> across
> > threads would then be segregated (it would also likely be @system in most cases). More complex objects can then have shared member functions which deal with all of the threading stuff internally rather than requiring
> that
> > everyone using them deal with it directly, but at the core, it's
> ultimately
> > going to be a question of atomics or casts if we want the guarantee that shared objects aren't being accessed across threads when they're not properly protected.
> >
> > However, the core problem with all of this is that the restrictions that need to be put on shared have never really been put in place. For years, about all that it really did was disallow implicit conversions between shared and thread-local (and even then, it didn't always do it - e.g.
> with
> > integers). Over time, additional restrictions have been added (I think
> > mostly to numeric types), but it's all piecemeal. IIRC, there was a DIP
> that
> > was supposed to outright make reading and writing to shared variables illegal, but I'm not quite sure where that stands. We also potentially
> need
> > some language improvements to make it cleaner, because otherwise in order to read or write value types, we'll have to do stuff like take their address and cast the resulting pointer so that we can operate on the
> object
> > as thread-local while the mutex is locked (or whatever or threading
> > protections are in place).
> >
> > So, while I think that the basic idea of shared is solid, it's never been properly implemented.
>
> LOL. And reading through the rest of the thread, I see that Manu already brought this up and that the DIP has an implementation with the -preview flag.
>
> - Jonathan M Davis


👍

You're right though! :P

On Sunday, 2 August 2020 at 20:50:14 UTC, Andrei Alexandrescu wrote:
>
> * Nobody - probably not even Timon - knows what "shared" does or is supposed to do and not do. The most I got from Walter ever is "shared is intentionally restricted so you can't do much without a cast". Yet the definition of "much" and the conditions under which casting is legit are not anywhere to be found.
>

Shouldn't shared just be a relaxation in compiler checks so that any data that has the "shared" qualifier is allowed to be shared between threads. This is pretty much what __gshared is about. For the compiler to check if you really can share the data is currently impossible with current compiler technology. It can be atomic integers, Intel TSX, some very elaborate lockless algorithm, lockless COW, even mutexes/spinlocks if necessary. Programmer must "manually" ensure that the data can be shared, there is no other way.

So yes, a revisit of the qualifiers should be done so that they can be precisely defined.

On Thursday, 13 August 2020 at 19:22:29 UTC, IGotD- wrote:
> Shouldn't shared just be a relaxation in compiler checks so that any data that has the "shared" qualifier is allowed to be shared between threads. This is pretty much what __gshared is about.

It should help the programmer write bug-free code, which __gshared does not. A panacea would be great, but more realistically we should disallow reading, writing and calling non-shared methods on shared data in @safe code. This limits the number of places where races can exist, but still (sadly) requires the programmer to know what he's doing when he writes shared code.

--
  Simen

On 8/4/20 12:57 PM, Timon Gehr wrote:
>> * Every time "inout" comes within a radius of a mile of what I'm doing, it starts to stink like a skunk. I wish I could get a restraining order. I can't instantiate "inout" variables, so writing any tests or constraints becomes an advanced matter of defining functions and crap.
> 
> I have never understood the `(inout int){ T.init; }` idiom. Just use `(T value){ value; }`.
> 
>> I get frustrated, I protest to this forum, and immediately a cabal is raised under Timon's leadership. The cabal convinces me that inout is actually great and that I'm an idiot. I do get convinced, which is more of a proof that Timon is very good, than a testament to the conviviality of inout. Then I leave and get back to my code, and it stinks of inout again. And I hate it and myself for having to deal with it.
>> ...
> 
> I am sure you are sincere, but I still think this is a misrepresentation. I don't think I ever claimed that `inout` is great. I merely understand what `inout` is supposed to be, but it comes way short. See all of the issues I have opened that show that type checking for `inout` is broken. When I tried to document inout properly in 2018 I found multiple new type system holes, I think they are open to this day.

Well for what it's worth I have a simple question: how can I assess in druntime if a type T is copyable? I add the informal requirement that it's a simple query so it should be served with a proportionally simple answer.

My initial take:

static if (is(typeof((T x) { T y = x; }))) { ... }

i.e. a lambda can be created that takes a T and creates a copy of it. Beautiful.

This test, however, passes for inout types. And inout types cannot be considered really copyable, because they cannot be used in many places where one would expect to use a copyable type. To wit, a variety of unittests will fail (such as structs with copyable members), all protesting to the attempt of classifying inout types as copyable.

Second attempt:

static if (is(typeof((T x) { T y = x; })) && !is(T == inout U, U) { ... }

So a type is copyable as before, just let's special case inout for exclusion.

This already gets my diaper in a bunch because I need to special case a type of which utility I already am suspicious. And it's not only here - it's many, many similar places.

Also, this also does NOT work because inout(const(int)) passes the test. This could probably be classified as a bug in the language or its compiler.

So now I'm looking at things like importing "core.lifetime : emplace" and see if that compiles. Because the very complex implementation of emplace uses a complex mechanism to handle inout.

I could be convinced that this awful complexity is justified given the choices made in the definition of this or that, but it would be more difficult to convince ourselves this is good programming language design. Simple questions should have simple answers.

On 8/14/20 9:45 AM, Andrei Alexandrescu wrote:

> Well for what it's worth I have a simple question: how can I assess in druntime if a type T is copyable? I add the informal requirement that it's a simple query so it should be served with a proportionally simple answer.
> 
> My initial take:
> 
> static if (is(typeof((T x) { T y = x; }))) { ... }
> 
> i.e. a lambda can be created that takes a T and creates a copy of it. Beautiful.
> 
> This test, however, passes for inout types. And inout types cannot be considered really copyable, because they cannot be used in many places where one would expect to use a copyable type. To wit, a variety of unittests will fail (such as structs with copyable members), all protesting to the attempt of classifying inout types as copyable.

inout types aren't any less copyable than const or immutable types.

> 
> Second attempt:
> 
> static if (is(typeof((T x) { T y = x; })) && !is(T == inout U, U) { ... }
> 
> So a type is copyable as before, just let's special case inout for exclusion.

This seems silly, if it's copyable it's copyable. The unrestricted test should be valid.

> 
> This already gets my diaper in a bunch because I need to special case a type of which utility I already am suspicious. And it's not only here - it's many, many similar places.
> 
> Also, this also does NOT work because inout(const(int)) passes the test. This could probably be classified as a bug in the language or its compiler.
> 
> So now I'm looking at things like importing "core.lifetime : emplace" and see if that compiles. Because the very complex implementation of emplace uses a complex mechanism to handle inout.
> 
> I could be convinced that this awful complexity is justified given the choices made in the definition of this or that, but it would be more difficult to convince ourselves this is good programming language design. Simple questions should have simple answers.

Is there a specific example? I suspect we are going to get into one of the foolish restrictions of inout -- can't be struct members or can't be returned if you don't have inout parameters.

The answer is still -- fix inout so it's not so foolish.

-Steve

On Sunday, 2 August 2020 at 20:50:14 UTC, Andrei Alexandrescu wrote:
>
> A "Define All Qualifiers" DIP would be a radical improvement of the state of affairs.

Make sure there is a separate DIP for each qualifier. I have a feeling some qualifiers will be quite a battle with different opinions.