On Tue, Oct 16, 2018 at 3:25 PM Nicholas Wilson via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On Tuesday, 16 October 2018 at 21:19:26 UTC, Steven Schveighoffer wrote:
> > There is in fact, no difference between:
> >
> > int *p;
> > shared int *p2 = p;
> > int *p3 = cast(int*)p2;
> >
> > and this:
> >
> > int *p;
> > shared int *p2 = p;
> > int *p3 = p;
>
> If I understand Manu correctly the first should compile, and the second should error, just like if you replaces shared with const in the above.

Why is the second an error?

Right, for the sake of the thought experiment, replace shared with const.
The first is an abomination (casting away const)... the second is just a no-op.

On Wednesday, 17 October 2018 at 00:29:04 UTC, Manu wrote:
> On Tue, Oct 16, 2018 at 3:25 PM Nicholas Wilson via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>
>> On Tuesday, 16 October 2018 at 21:19:26 UTC, Steven Schveighoffer wrote:
>> > There is in fact, no difference between:
>> >
>> > int *p;
>> > shared int *p2 = p;
>> > int *p3 = cast(int*)p2;
>> >
>> > and this:
>> >
>> > int *p;
>> > shared int *p2 = p;
>> > int *p3 = p;
>>
>> If I understand Manu correctly the first should compile, and the second should error, just like if you replaces shared with const in the above.
>
> Why is the second an error?
>
> Right, for the sake of the thought experiment, replace shared with const.
> The first is an abomination (casting away const)... the second is just a no-op.

I missed that the third example was *p3 = p; not *p3 = p2;

On Tuesday, 16 October 2018 at 06:21:22 UTC, Manu wrote:
> On Mon, Oct 15, 2018 at 8:55 PM Isaac S. via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>
>> On Tuesday, 16 October 2018 at 02:26:04 UTC, Manu wrote:
>> >> I understand your point but I think the current shared (no implicit conversion) has its uses.
>> >> *snip*
>> >
>> > If you can give a single 'use', I'm all ears ;)
>>
>> My usages are a custom ref counted template and list types (which are built on top of the former). The ref counted template will initialize naively when not shared but utilize compare-and-set and thread yielding if needed when shared (ensureInitialized can occur any time after declaration). The list types (which are not shared-compatible *yet*) will utilize a read-write mutex only when shared (no cost beyond a little unused memory to non-shared objects). As such, casting any of the non-shared versions of these types to shared would be unsafe.
>>
>> (Technically the types can be safely casted to shared so long as no non-shared reference to them exists and vice versa.)
>
> Can you link to code? It doesn't sound incompatible with my proposal. Mutexes are blunt instruments, and easy to model. My whole suggestion has virtually no effect on the mutex protected case, which is what most people seem to talk about.

On Wednesday, 17 October 2018 at 00:26:58 UTC, Manu wrote:
> *snip*
>
> Overloading for shared and unshared is possible, and may be desirable in many cases.
> There are also many cases where the code duplication and tech-debt
> does not carry its weight. It should not be required, because it's not technically required.

Overloading for shared and unshared is my reason for not allowing implicit conversion on my types (I have no problems with implicit conversion being optional or disableable). The unshared function performs no synchronization of any kind while the shared function always does. This means that having an unshared and shared reference to the same object is unsafe.

As for an actual link to code, I can really only link to my ref counted template
as I haven't gotten around to making the containers shared-compatible yet. The main point of interest is at lines 103-131:

https://github.com/isaacs-dev/Familiar/blob/66f1a94fc099601465e755d40a2c68bf4200cabd/containers/familiar/containers/safe_ref_counted.d#L103

The unshared ensureInitialized() doesn't worry about threads simultaneously calling it. The shared version of it uses compare-and-set to prevent two threads from initializing it at the same time. An unsafe example would require a shared reference/pointer to an unshared SafeRefCounted (which would be fairly weird to do) and both calling ensureInitialized. While an example using the ref counted template is fairly contrived, a list-type that only uses a read-write mutex when it's shared isn't.


As to the larger part of preventing reading/writing of shared objects, I agree with its purpose and have no problems with it.

On Tue, Oct 16, 2018 at 8:20 PM Isaac S. via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On Tuesday, 16 October 2018 at 06:21:22 UTC, Manu wrote:
> > On Mon, Oct 15, 2018 at 8:55 PM Isaac S. via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>
> >> On Tuesday, 16 October 2018 at 02:26:04 UTC, Manu wrote:
> >> >> I understand your point but I think the current shared (no
> >> >> implicit conversion) has its uses.
> >> >> *snip*
> >> >
> >> > If you can give a single 'use', I'm all ears ;)
> >>
> >> My usages are a custom ref counted template and list types (which are built on top of the former). The ref counted template will initialize naively when not shared but utilize compare-and-set and thread yielding if needed when shared (ensureInitialized can occur any time after declaration). The list types (which are not shared-compatible *yet*) will utilize a read-write mutex only when shared (no cost beyond a little unused memory to non-shared objects). As such, casting any of the non-shared versions of these types to shared would be unsafe.
> >>
> >> (Technically the types can be safely casted to shared so long as no non-shared reference to them exists and vice versa.)
> >
> > Can you link to code? It doesn't sound incompatible with my proposal. Mutexes are blunt instruments, and easy to model. My whole suggestion has virtually no effect on the mutex protected case, which is what most people seem to talk about.
>
> On Wednesday, 17 October 2018 at 00:26:58 UTC, Manu wrote:
> > *snip*
> >
> > Overloading for shared and unshared is possible, and may be
> > desirable in many cases.
> > There are also many cases where the code duplication and
> > tech-debt
> > does not carry its weight. It should not be required, because
> > it's not technically required.
>
> Overloading for shared and unshared is my reason for not allowing implicit conversion on my types (I have no problems with implicit conversion being optional or disableable). The unshared function performs no synchronization of any kind while the shared function always does. This means that having an unshared and shared reference to the same object is unsafe.

Okay, so just to be clear, you're objecting to an immensely useful
behaviour because you exploit the current design as an optimisation
potential?
That said, I'm still not taking anything away from you... you can
still implement your class that way if you like, no change will affect
that decision.

You obviously have mechanisms in place to guarantee the
thread-local-ness of your object (otherwise it wouldn't work), so,
assuming you implement the unshared overload to be unsafe, then
situation hasn't changed for you at all...?
I don't think a robust middleware would make that trade-off, but an
application can do that if it wishes. Trading implicit safety
(enforcing it externally via application context) for perf is not
unusual at all.

> As to the larger part of preventing reading/writing of shared objects, I agree with its purpose and have no problems with it.

Right, the implicit cast thing is secondary to this fundamental part.
I think it's worth first focusing on getting that rule right.
shared = no read + no write .. I don't think that's objectionable.

On Wednesday, 17 October 2018 at 03:50:44 UTC, Manu wrote:
> On Tue, Oct 16, 2018 at 8:20 PM Isaac S. via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>> *snip*
>>
>> Overloading for shared and unshared is my reason for not allowing implicit conversion on my types (I have no problems with implicit conversion being optional or disableable). The unshared function performs no synchronization of any kind while the shared function always does. This means that having an unshared and shared reference to the same object is unsafe.
>
> Okay, so just to be clear, you're objecting to an immensely useful
> behaviour because you exploit the current design as an optimisation potential?
> That said, I'm still not taking anything away from you... you can still implement your class that way if you like, no change will affect that decision.
>
> You obviously have mechanisms in place to guarantee the
> thread-local-ness of your object (otherwise it wouldn't work), so, assuming you implement the unshared overload to be unsafe, then
> situation hasn't changed for you at all...?
> I don't think a robust middleware would make that trade-off, but an application can do that if it wishes. Trading implicit safety
> (enforcing it externally via application context) for perf is not unusual at all.

I'm not objecting to the behavior (I actually really want it in the language as it allows for some really useful designs) but more-so saying it would be nice if it could be enabled or disabled. If its enabled-by-default, the @disable attribute could be used somehow (maybe "@disable alias this shared;").

Even without the ability to enable/disable shared implicit conversion it won't break my code, it'll just allow a potential bug. Since it does allow for extremely useful designs, I'll support this change (*even if it can't be enabled/disabled*).

On 10/15/2018 11:46 AM, Manu wrote:
> [...]

Shared has one incredibly valuable feature - it allows you, the programmer, to identify data that can be accessed by multiple threads. There are so many ways that data can be shared, the only way to comprehend what is going on is to build a wall around shared data.

(The exception to this is immutable data. Immutable data does not need synchronization, so there is no need to distinguish between shared and unshared immutable data.)

This falls completely apart if shared and unshared data can be aliased. When Andrei and I came up with the rules for:

   mutable
   const
   shared
   const shared
   immutable

and which can be implicitly converted to what, so far nobody has found a fault in those rules. Timon Gehr has done a good job showing that they still stand unbreached.

So, how can mutable data become shared, and vice versa? I've never found a way to do that that is proveably safe. Therefore, it's up to the programmer.

If a programmer does the following:

    T* p;
    shared(T)* sp;

    p = cast(T*)sp;           (1)
    sp = cast(shared(T)*) p;  (2)

those casts will be rejected in @safe code. They'll have to be in @trusted or @system code. It will be up to the programmer to ensure (via mutexes, locks, uniqueness, etc.) that:

    (1) sp is a unique pointer and that ownership of the data is thus transferred for the lifetime of p

    (2) p is a unique pointer and that ownership of the data is thus transferred for the lifetime of sp

A sensible program should try to minimize the places where there are these "gates" through the wall, as those gates will be where you'll be looking when threading bugs appear.

Allowing implicit aliasing between shared and unshared data means the entire program needs to reviewed for threading bugs, rather than just the shared parts. One might as well just not have shared as a language feature at all. D being a systems programming language, one can certainly write code that way, just like one does with C, and with all of the threading bugs one gets doing that.

----------

Currently, the compiler allows you to read/write shared data without any locks or atomics. I.e. the compiler doesn't attempt to insert any synchronization on your behalf. Given all the ways synchronization can be done, it just seems presumptive and restrictive to impose a particular scheme. It's pretty much left up to the user.

Which is why I recommend minimizing the amount of code that actually has to deal with shared data.

On Tue, Oct 16, 2018 at 10:45 PM Walter Bright via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 10/15/2018 11:46 AM, Manu wrote:
> > [...]
>
> Shared has one incredibly valuable feature - it allows you, the programmer, to identify data that can be accessed by multiple threads. There are so many ways that data can be shared, the only way to comprehend what is going on is to build a wall around shared data.
>
> (The exception to this is immutable data. Immutable data does not need synchronization, so there is no need to distinguish between shared and unshared immutable data.)
>
> This falls completely apart if shared and unshared data can be aliased.

What does it mean 'aliased' precisely? I'm trying to prevent that in
practical terms.
If you can't read/write to a shared object... is it actually aliased?
Or do you just have a fancy int that happens to look like a pointer?

> When Andrei and I came up with the rules for:
>
>     mutable
>     const
>     shared
>     const shared
>     immutable
>
> and which can be implicitly converted to what, so far nobody has found a fault in those rules.

Oh bollocks... everyone has been complaining about this for at least
the 10 years I've been here!
Shared is uninteresting and mostly useless as spec-ed, everyone knows this.
Interaction with shared via barely-controlled blunt casting in
@trusted blocks is feeble and boring. It doesn't really give us
anything in practice that we don't have in C++. It's an uninspired
design.

I am working against a design where *everything* may be shared, and
it's a very interesting design space. I have no doubt it represents
the future of my field.
We are limited by what the type system can express on this front. I'm
trying to explore opportunities to make shared interesting and
useful... and I'm mostly being met with prejudice here.
Eject what you think you know about shared from your brain, and try
and take a fresh look from the perspective I'm proposing.
I think you'll find that the current wisdom and interactions with
shared are actually preserved in practice, but new opportunities
become available. If not, I want to understand how the design is
broken, so I can continue to iterate.

If D offered a real advantage here over C++, this would really represent a strong attracting force.

> Timon Gehr has done a good job showing that they still stand unbreached.

His last comment was applied to a different proposal.
His only comment on this thread wasn't in response to the proposal in
this thread.
If you nominate Timon as your proxy, then he needs to destroy my
proposal, or at least comment on it, rather than make some prejudiced
comment generally.

> So, how can mutable data become shared, and vice versa? I've never found a way to do that that is proveably safe. Therefore, it's up to the programmer.

I'm proposing a way, and that is:
1. the rule must be applied that shared object can not be read or written
2. attributing a method shared is a statement and a promise that the
method is threadsafe

The rest just follows naturally.

> If a programmer does the following:
>
>      T* p;
>      shared(T)* sp;
>
>      p = cast(T*)sp;           (1)
>      sp = cast(shared(T)*) p;  (2)
>
> those casts will be rejected in @safe code. They'll have to be in @trusted or @system code. It will be up to the programmer to ensure (via mutexes, locks, uniqueness, etc.) that:
>
>      (1) sp is a unique pointer and that ownership of the data is thus
> transferred for the lifetime of p
>
>      (2) p is a unique pointer and that ownership of the data is thus
> transferred for the lifetime of sp

'Transfer' is a nice idea, but it's not what's happening here. That's
not what these operations do.
Transferring ownership is a job for move semantics, and that's nowhere in sight.

Also, there's no need to 'transfer' a this pointer to shared to call a threadsafe method. You can always just call it safely.

> A sensible program should try to minimize the places where there are these "gates" through the wall, as those gates will be where you'll be looking when threading bugs appear.

I agree, that's my goal here. Shared right now is an unsafe mess; I want to apply aggressive restriction so that you can't do unsafe operations without explicit casts.

> Allowing implicit aliasing between shared and unshared data means the entire program needs to reviewed for threading bugs, rather than just the shared parts.

That's backwards, I'm suggesting conversion TO shared, such that you
are able to call threadsafe methods on thread-local things. There's no
possible harm in that.
I'm trying to eliminate aliasing by removing read/write access.
The result is, the only thing you are able to do with a shared
reference are explicitly threadsafe things. Any other operation is
inaccessible without deliberate blunt casting, and the rules
surrounding that are the same as always; those that you listed above.

> One might as well just not have shared as a language feature at all. D being a systems programming language, one can certainly write code that way, just like one does with C, and with all of the threading bugs one gets doing that.

And this is a silly claim that's been made before in this thread. It
doesn't make sense, and I kinda have to presume here you have either
not read or do not understand my proposal...
Are you suggesting that I hate shared, and I want to make it
worthless, therefore I'm trying to change the rules to eliminate it
from practical existence?
I promise you, I'd *really* like it if shared were a _useful_ language
feature. The suggestion that I intend to undermine it such that it not
be a language feature at all is obviously ridiculous.


> ----------
>
> Currently, the compiler allows you to read/write shared data without any locks or atomics.

Right, this is the core of the problem, and it must change for shared to model anything useful.

> I.e. the compiler doesn't attempt to insert any synchronization on
> your behalf. Given all the ways synchronization can be done, it just seems
> presumptive and restrictive to impose a particular scheme. It's pretty much left
> up to the user.

But it's a useless model; it's just a boring sentinel tag with no
functional application. The shared rules don't model any useful
behaviour.
I'm trying to express how shared could model a useful behaviour, and
coincidentally be *more* conceptually safe, and intuitively model and
communicate appropriate interactions with threadsafe objects.

> Which is why I recommend minimizing the amount of code that actually has to deal with shared data.

Well, the future is SMP. 100% of our code potentially deals with
shared data, and it would be idea to do so in a typesafe manner.
We can't model this with C++. We can't model it in D either, because
D's `shared` doesn't model anything... but with just the one change
that shared can't read or write, we would have the foundation of
something useful.
Being able to read/write to shared objects is weird; it's completely
unsafe in every event. It's a useless access right, and changing it
such that shared can not read/write would improve the model in such a
way that it starts to become useful.

On Wednesday, 17 October 2018 at 05:40:41 UTC, Walter Bright wrote:

> When Andrei and I came up with the rules for:
>
>    mutable
>    const
>    shared
>    const shared
>    immutable
>
> and which can be implicitly converted to what, so far nobody has found a fault in those rules...

Here's one: shared -> const shared shouldn't be allowed. Mutable aliasing in single-threaded code is bad enough as it is.

On Wednesday, 17 October 2018 at 05:40:41 UTC, Walter Bright wrote:
> On 10/15/2018 11:46 AM, Manu wrote:
>> [...]
>
> Shared has one incredibly valuable feature - it allows you, the programmer, to identify data that can be accessed by multiple threads. There are so many ways that data can be shared, the only way to comprehend what is going on is to build a wall around shared data.
>
> (The exception to this is immutable data. Immutable data does not need synchronization, so there is no need to distinguish between shared and unshared immutable data.)
> [snip]


Isn't that also true for isolated data (data that only allows one alias)?

On Wednesday, 17 October 2018 at 07:20:20 UTC, Manu wrote:
> [snip]
>
> Oh bollocks... everyone has been complaining about this for at least
> the 10 years I've been here!
> [snip]

As far as I had known from reading the forums, shared was not feature complete.

Also, are you familiar with Atila's fearless library for safe sharing?
https://github.com/atilaneves/fearless