On Tuesday, 4 August 2020 at 12:52:01 UTC, Manu wrote:
> On Tue, Aug 4, 2020 at 5:40 PM Sebastiaan Koppe via Digitalmars-d < digitalmars-d@puremagic.com> wrote:
>> There is just one thing about shared I don't understand. If I design my object such that the non-shared methods are to be used thread-local and the shared methods from any thread, it follows that I should be able to call said shared methods from both a shared and non-shared instance of that object.
>
> Yes, this is a thing I talked at great length 1-2 years ago.
> If you take shared to mean "is thread-safe", then my idea was that
> not-shared -> shared implicit conversion should be possible.
> What I often do is this:
>
> struct Thing
> {
>   ref shared(Thing) implSharedCast() { return *cast(shared)&this; }
>   alias implSharedCast this;
> }
>
> If that were an implicit conversion, that implies a slight change of
> meaning of shared (to one that I consider immensely more useful), but it's
> more challenging for the compiler to prove with confidence, and there's a
> lot of resistance to this change.

What exactly does the compiler need to prove? The restrictions are all in place, you can only call shared methods on a shared object, and you can only access shared members in a shared method.

Hmm, well I guess if members themselves were also implicitly promoted to shared that could cause havoc.

> In the mean-time, until the shared usage patterns are more well-proven, I
> recommend you try to use the 'alias this' pattern I show above, and report
> on any issues you encounter using this scheme. If no issues are identified
> with extensive real-world usage data, I will push again for that implicit
> conversion rule.

Great, I'll try it. Thanks.

General disclaimer: My post may contain rebuttals to low-level technical points which were made in support of some high-level claims. Apparently there has been some confusion: When I disagree with a technical point, I am not automatically taking a stance against the corresponding high-level claim.

On 02.08.20 22:50, Andrei Alexandrescu wrote:
> (Background: qualifiers were introduced following my horror when I started writing D1 code and saw that strings are represented as char[]. So structs with string members would look like:
> 
> struct Widget
> {
>      int x, y;
>      char[] name, parentName;
>      ...
> }
> 
> I found it just shocking that following a Widget's construction whoever aliased the same strings the outside could change members of the Widget without Widget knowing. Of course, other D1 coders disliked that as well, so they'd defensively duplicate in the constructor:
> 
> struct Widget
> {
>      int x, y;
>      char[] name, parentName;
>      this(char[] n, char[] p)
>      {
>          name = n.dup;
>          parentName = p.dup;
>      }
>      ...
> }
> 
> thus ensuring proliferation of the garbage whether duplication was needed or not.
> 
> I found this absolutely maddening, to the extent I didn't think D could be ever used at any considerable scale while dragging this anchor behind it.
> 
> The second problem was Walter was adamant about using arrays of characters at strings. He found the notion of a library-defined string type (a la C++) an absolute abomination. Stubborn about it like I've never seen him before or after. So my unstoppable requests for a string type were met with the proverbial immovable refusal. Ironically, much of his argument came from an efficiency angle, yet the unnecessary duplication was way less efficient than some reference counting/small string optimization/etc scheme that a dedicated string type would use. ...

At the cost of efficient slicing?

> Then we figured things would work out if we arranged things such that
> people could NOT change individual characters of a string. That would
> allow sharing without the danger of long-distance influence. After many
> discussions with Walter, Bartosz, Eric, Brad, and myself, immutable and
> const were born.
> ...

And it has to be said that they do a great job at preventing individual chars in a `char[]` from being mutated. :-)

> Then followed the other qualifiers, in order: shared and inout.)
> 
> * * *
> 
> The result is... there: https://dlang.org/spec/const3.html. It has the
> images https://dlang.org/images/qualifier-combinations.svg and
> https://dlang.org/images/qualifier-conversions.svg and a large table and
> a lot of rules. Whenever I code anything generic, I find myself going
> back to those damn images and tables way more than anyone ought to. (And
> it's ironic... I made those. Woe to the relative newcomer.)
> ...

Well, those tables and figures are easy to make, because they are based on simple rules. The combinatorial explosion is artificial.

> It's all too complicated, making generic D programming into 3D chess
> instead of the difficult endeavor it already is. And what does it buy
> us? Well, we don't need to define a string library type. Yowzers.
> (Actually we should if we want to get rid of the GC. But then Walter
> would oppose that. So - stalemate once again.)
> ...

In general, Walter opposes memory ownership to be mediated by library types.

> Far as I can tell, the power/weight ratio of qualifier is very poor. I wish a DIP would revisit qualifiers with a stated intent to simplify them as much as possible. Whenever I code generically I invariably run into these issues:
> 
> * Whatever I do, however I twitch, immutable finds the opportunity to
> lodge itself in a soft part of my body and cause constant pain. This
> doesn't work, that doesn't work. No solution for "tail immutable" -
> mutable references to immutable class instances can't be done without
> contortions. Can't assign to out immutable class references, though
> there's no reason for that (see Adam's recent post).
> ...

Personally I don't use it where it causes those problems, but maybe the standard library is not afforded those conveniences.

> * No matter how I dice any significant piece of code, there will be five
> casts from immutable and/or back that I can't rid of and lose sleep at
> night trying to convince myself are justified.
> ...

Maybe `immutable` is overused in that code base.

> * Every time "inout" comes within a radius of a mile of what I'm doing, it starts to stink like a skunk. I wish I could get a restraining order. I can't instantiate "inout" variables, so writing any tests or constraints becomes an advanced matter of defining functions and crap.

I have never understood the `(inout int){ T.init; }` idiom. Just use `(T value){ value; }`.

> I get frustrated, I protest to this forum, and immediately a cabal is
> raised under Timon's leadership. The cabal convinces me that inout is
> actually great and that I'm an idiot. I do get convinced, which is more
> of a proof that Timon is very good, than a testament to the conviviality
> of inout. Then I leave and get back to my code, and it stinks of inout
> again. And I hate it and myself for having to deal with it.
> ...

I am sure you are sincere, but I still think this is a misrepresentation. I don't think I ever claimed that `inout` is great. I merely understand what `inout` is supposed to be, but it comes way short. See all of the issues I have opened that show that type checking for `inout` is broken. When I tried to document inout properly in 2018 I found multiple new type system holes, I think they are open to this day.

I'm attaching my draft write-up on `inout` from 2018.

> * Nobody - probably not even Timon - knows what "shared" does or is
> supposed to do and not do. The most I got from Walter ever is "shared is
> intentionally restricted so you can't do much without a cast". Yet the
> definition of "much" and the conditions under which casting is legit are
> not anywhere to be found.
> ...

Well, I think I know what it is supposed to do, but I think Walter does not fully agree, as evidenced by the `scope shared` discussion. However, think of it this way: it is clear what _unshared_ does; it's a type system assertion that there will not be concurrent accesses to the given memory location. Casts to and from shared have to uphold that. That's essentially it. Then, semantically, shared variables are just standard variables from C or C++. Of course, that means `shared` variables should not be accessed using unsynchronized reads/writes in `@safe` functions, but I think Manu and Walter have been working on adding necessary type checks.

> * And of course, "shared" gladly partakes in qualifier combinations, thus spreading its stink around in a combinatorial manner.

That's true of other language features. "Any field of any type can be of type `int`, so `int` is spreading its stink around in a combinatorial manner."

If you consider type constructors to be type constructors instead of flags on top of a type, I think they are easier to understand and you don't have to handle nearly the same amount of combinatorial explosion.

> Believe it or
> not, the type "const inout shared T" exists. Of course, nobody knows
> what it really means or how it could be used.
> ...

I beg to differ. It's either const(shared(T)) or immutable(T), but you don't know which. An use case is you have a function that may return references to its `shared` input, which it does not modify, and you want to preserve `immutable` qualifiers across function calls while also allowing the function to be called with mutable shared data.

Unfortunately this is only obvious if you understand `inout`, but unfortunately, `inout` is harder to understand than type theory concepts that are more general than `inout`...

> A "Define All Qualifiers" DIP would be a radical improvement of the state of affairs.

My attached draft has some overlap with that.

On 04.08.20 04:13, Andrei Alexandrescu wrote:
> On 8/3/20 5:56 PM, Manu wrote:
>> I think shared has potential to offer critical benefit to D, and we've talked about this personally to some length. Don't kill it yet, let's try and fix it. Although it's worth recognising that if we don't fix it, then it might as well be killed as it stands today.
> 
> Of course fixing it would be great!

It's being fixed, but Manu's vision for "fixing" it is really to do more than fixing it.  Manu thinks "unshared" is useless as a type qualifier.

He wants to change the meaning of `shared` so it no longer means "shared", but instead means "thread safe". For plain variables, this "thread safe" annotation would also imply "shared" (in an unprincipled manner, breaking the type system).

"thread safe" is to "shared" approximately as "const" is to "mutable". That's also why all variables can implicitly convert to "thread safe".

> I'm glad you pushed the restriction through. At least now generic code could see `shared` as "shrouded in opacity, not subject to the usual operations".

Luckily the restriction makes sense for `shared` even if its meaning is not changed to be different from its name.

On 04.08.20 14:52, Manu wrote:
> slight change of meaning of shared

It's not slight. What happens to module level variables?

On 04.08.20 15:52, Sebastiaan Koppe wrote:
> On Tuesday, 4 August 2020 at 12:52:01 UTC, Manu wrote:
>> On Tue, Aug 4, 2020 at 5:40 PM Sebastiaan Koppe via Digitalmars-d < digitalmars-d@puremagic.com> wrote:
>>> There is just one thing about shared I don't understand. If I design my object such that the non-shared methods are to be used thread-local and the shared methods from any thread, it follows that I should be able to call said shared methods from both a shared and non-shared instance of that object.
>>
>> Yes, this is a thing I talked at great length 1-2 years ago.
>> If you take shared to mean "is thread-safe", then my idea was that
>> not-shared -> shared implicit conversion should be possible.
>> What I often do is this:
>>
>> struct Thing
>> {
>>   ref shared(Thing) implSharedCast() { return *cast(shared)&this; }
>>   alias implSharedCast this;
>> }
>>
>> If that were an implicit conversion, that implies a slight change of
>> meaning of shared (to one that I consider immensely more useful), but it's
>> more challenging for the compiler to prove with confidence, and there's a
>> lot of resistance to this change.
> 
> What exactly does the compiler need to prove? The restrictions are all in place, you can only call shared methods on a shared object, and you can only access shared members in a shared method.
> ...

Nope. You can access all members, but they will be treated as `shared`.

> Hmm, well I guess if members themselves were also implicitly promoted to shared that could cause havoc.

Which is what the code above does, and also what Manu's "thread safe" qualifier would do. `shared` would no longer mean "shared".

On 8/4/20 12:57 PM, Timon Gehr wrote:
> My attached draft has some overlap with that.

Thanks, Timon. Mike, looks like we have a great post for the blog in the making!

Alternative syntax (even though it is in the template parameters, it is in fact erased and should not end in duplication):






T(U) first(void T=inout, U)(T(U)[] data) {
	assert(data.length > 0);
	return data[0];
}

--------

struct Foo(void Qual=inout) {
	private int[] _payload;
	
	this(Qual int[] payload) void(Qual) {
		this._payload = payload;
	}
	
	@property Qual(int)[] payload() void(Qual) {
		return this._payload;
	}
}

--------

void assign(void Writable=inout, void Readable=inout)(ref Writable(int)* a,Readable(int)* b,ref Writable(int)* c,Readable(int)* d) {
    a=b;
    c=d;
}

--------

Qual(int)* id(void Qual=inout)(Qual(int)* p) {
    return p;
}

--------

static assert(is(typeof(&id!mutable)==int* function(int*)))
static assert(is(typeof(&id!immutable)==immutable(int)* function(immutable(int)*)))
static assert(is(typeof(&id!const)==const(int)* function(const(int)*)))

That random mutable identifier tho...

On Tue, Aug 4, 2020 at 11:55 PM Sebastiaan Koppe via Digitalmars-d < digitalmars-d@puremagic.com> wrote:

> On Tuesday, 4 August 2020 at 12:52:01 UTC, Manu wrote:
> > On Tue, Aug 4, 2020 at 5:40 PM Sebastiaan Koppe via Digitalmars-d < digitalmars-d@puremagic.com> wrote:
> >> There is just one thing about shared I don't understand. If I design my object such that the non-shared methods are to be used thread-local and the shared methods from any thread, it follows that I should be able to call said shared methods from both a shared and non-shared instance of that object.
> >
> > Yes, this is a thing I talked at great length 1-2 years ago.
> > If you take shared to mean "is thread-safe", then my idea was
> > that
> > not-shared -> shared implicit conversion should be possible.
> > What I often do is this:
> >
> > struct Thing
> > {
> >   ref shared(Thing) implSharedCast() { return
> > *cast(shared)&this; }
> >   alias implSharedCast this;
> > }
> >
> > If that were an implicit conversion, that implies a slight
> > change of
> > meaning of shared (to one that I consider immensely more
> > useful), but it's
> > more challenging for the compiler to prove with confidence, and
> > there's a
> > lot of resistance to this change.
>
> What exactly does the compiler need to prove? The restrictions are all in place, you can only call shared methods on a shared object, and you can only access shared members in a shared method.
>

Marking the function shared (and therefore the data accessible) doesn't magically make the function body threadsafe; what it does it makes the function potentially racy, and subject to VERY careful implementation. Sadly, I'm not aware of any CS research that can help you write atomic/threadsafe functions with any degree of proof in an environment like this (ie, without isolation and/or things like copy-on-write, etc).

Calling shared methods from shared methods isn't safe either. Each call may be threadsafe atomically, but you can't author a leaf function without just as much care (or more) than the lower-level ones. If the function is more than 1-line, and carries some state across a few statements that call lower-level shared methods, then you need to be confident about the atomic state guarantees (or not! which is more likely) of the dependency API's, such that you don't create a state race between calls.

Needless to say, it's still tricky, and there's really nothing the language
can help you do... other than provide a strong mechanism to lock it off
from normal code.
As long as shared methods are few, and very well defined, then it is
possible to implement very interesting and useful machines with this
scheme, and also MASSIVELY help with validation of your ecosystem, and I
have done so... but it's not magic, it's just a lot better than C++.

The thing `shared` protects most against, is failure to correct multithreaded code when refactoring. I maintain a majorly-superscalar engine, and 90% of the race bugs we have had to spend heaps of time chasing down are the result of refactors or changes that had very peripheral contact with the multithreaded core; contact was narrow enough that we didn't notice then making changes, and there was nothing in the type system to alert us. For this reason alone, `shared` is worth its weight in gold.

Hmm, well I guess if members themselves were also implicitly
> promoted to shared that could cause havoc.
>

The catch though, is that if you allow implicit conversion from
unshared->shared, then the rule changes from "shared methods must be
threadsafe", to " shared methods must be threadsafe, AND unshared methods
must also be threadsafe in the event they are called in conjunction with
any of the shared methods (just not with eachother)".
While I have plenty of small tools where the second rule is easy to
implement, I can also think of many situations where even the first rule is
hard to implement, and the second rule is virtually impossible.

Unless we learn better ways to handle this, I don't think implicit conversion that way can scale. For the time being, you can implement the implicit conversion for the tools that may support that using `alias this` like I showed above, and I think that's appropriate for the time being.

In general, I'd like to find ways we can enhance shared to be more than a marker; consider TSAN (ThreadSanitizer) in Clang; it instruments atomics with runtime tracking to detect races at runtime. I think it would be possible to instrument shared data/methods in a similar way in debug builds, and then we'd have meaningful compiler assistance.

On Wed, Aug 5, 2020 at 3:20 AM Timon Gehr via Digitalmars-d < digitalmars-d@puremagic.com> wrote:

> On 04.08.20 04:13, Andrei Alexandrescu wrote:
> > On 8/3/20 5:56 PM, Manu wrote:
> >> I think shared has potential to offer critical benefit to D, and we've talked about this personally to some length. Don't kill it yet, let's try and fix it. Although it's worth recognising that if we don't fix it, then it might as well be killed as it stands today.
> >
> > Of course fixing it would be great!
>
> It's being fixed, but Manu's vision for "fixing" it is really to do more than fixing it.  Manu thinks "unshared" is useless as a type qualifier.
>
> He wants to change the meaning of `shared` so it no longer means "shared", but instead means "thread safe".


This's a highly speculative end-goal, but a lot of my work applies to
'shared' equally with respect to today's definition, and also that
potential definition.
Leaving the 'thread-safe' definition aside, the key goal I want to achieve
is that you may perform an implicit conversion to `scope shared`; since it
can't escape the callee and therefore no references should co-exist with
the thread-local instance in the calling scope.

This would be immensely useful to enable various parallel workloads,
parallel-for is a key target.
The trouble is, and I think you pointed it out before, when multiple
arguments alias eachother:

void fun(ref X a, ref scope shared(X) b) { /* a and b are a shared +
unshared alias... */ }

X x;
fun(x, x);

Allowing implicit conversion to `scope shared` requires that the caller and callee's scopes are a clear division between the unshared and the promoted-to-shared instances. scope enforces that we can't escape the callee's promoted-to-shared ref to alias the unshared ref in the callers scope, but that division can be violated by aliasing the caller's reference, and sneaking an unshared reference into the callee beside the promoted reference.

That problem makes the implicit conversion in the presence of scope challenging, and that is what leads to the design where `shared` comes to mean 'thread-safe' instead of simply shared; which insists a stronger set of requirements, and under that changed definition, it allows the alias to exist.

If there was another way to prevent the caller's unshared ref from aliasing its way into the callee, that would be preferable to the scheme I described before.

For plain variables, this
> "thread safe" annotation would also imply "shared" (in an unprincipled manner, breaking the type system).
>
> "thread safe" is to "shared" approximately as "const" is to "mutable". That's also why all variables can implicitly convert to "thread safe".
>
> > I'm glad you pushed the restriction
> > through. At least now generic code could see `shared` as "shrouded in
> > opacity, not subject to the usual operations".
>
> Luckily the restriction makes sense for `shared` even if its meaning is not changed to be different from its name.
>

It's not really 'lucky', it's fundamental to any conceivable definition of shared. It's a starting point that's actually possible to work out from.

On Wed, Aug 5, 2020 at 3:35 AM Timon Gehr via Digitalmars-d < digitalmars-d@puremagic.com> wrote:

> On 04.08.20 14:52, Manu wrote:
> > slight change of meaning of shared
>
> It's not slight. What happens to module level variables?
>

I'm not sure what you're asking?