Hi,

It's starting to get outright embarrassing to talk to newcomers about D's concurrency support because the most fundamental part of it -- the shared type qualifier -- does not have well-defined semantics at all.

I'm certainly not alone in being annoyed by this state of affairs: http://d.puremagic.com/issues/show_bug.cgi?id=8993

I've posted rants about the state of shared before and, from the comments on those, it appears that what most people want shared to do is at least one (and usually multiple) of

* make variables global (if appropriate in the context);
* make the wrapped type completely separate from the unwrapped type;
* make all operations be atomic;
* make all operations result in memory barriers.

At a glance, this looks fine. Exactly what you would want for shared types in a concurrent setting, right?

Except, not really. I'll try to explain all of the unsolved problems with shared below...

First of all, the fact that shared(T) is completely separate from T (i.e. no conversions allowed, except for primitive types) is a huge usability problem. In practice, it means that 99% of the standard library is unusable with shared types. Hell, even most of the runtime doesn't work with shared types. I don't know how to best solve this particular problem; I'm just pointing it out because anyone who tries to do anything non-trivial with shared will invariably run into this.

Second, the idea of making shared insert atomic operations is an absolute fallacy. It only makes sense for primitive types for the most part, and even for those, what sizes are supported depends on the target architecture. A number of ideas have come up to solve this problem:

* We make shared(T) not compile for certain Ts depending on the target architecture. I personally think this is a terrible idea because most code using shared will not be portable at all.
* We require any architecture D targets to support atomic operations for a certain size S at the very least. This is fine for primitives up to 64 bits in size, but doesn't clear up the situation for larger types (real, complex types, cent/ucent, ...).
* We make shared not insert atomic operations at all (thus making it kind of useless for anything but documentation).
* (Possibly others I have forgotten; please let me know if this is the case.)

I don't think any of these are particularly attractive, to be honest. If we do make shared insert atomic operations, we would also have to consider the memory ordering of those operations.

Third, we have memory barriers. I strongly suspect that this is a misnomer in most cases where people have suggested this; it's generally not useful to have a compiler insert barriers because they are used to control ordering of load/store operations which is something the programmer will want to do explicitly. In any case, the compiler can't usefully figure out where to put barriers, so it would just result in really bad performance for no apparent gain.

Fourth, there is implementation complexity. If shared is meant to insert specialized instructions, it will result in effectively two code paths for most code generation in any D compiler (read: maintenance nightmare).

Fifth, it is completely unclear whether casting to and from shared is legal (but with a big fat "caution" sign like casting away const) or if it's undefined behavior. Making it undefined behavior would further increase the usability problem I described above.

And finally, the worst part of all of this? People writing code that uses shared today are blindly assuming it actually does the right thing. It doesn't. Their code will break on any non-x86 platform. This is an absolutely horrifying situation now that ARM, MIPS, and PowerPC are starting to become viable targets for D.

Something needs to be done about shared. I don't know what, but the current situation is -- and I'm really not exaggerating here -- laughable. I think we either need to just make it perfectly clear that shared is for documentation purposes and nothing else, or, figure out an alternative system to shared, because I don't see shared actually being useful for real world work no matter what we do with it.

-- 
Alex Rønne Petersen
alex@lycus.org
http://lycus.org

On 11/11/12, Alex Rønne Petersen <alex@lycus.org> wrote:
> And finally, the worst part of all of this? People writing code that uses shared today are blindly assuming it actually does the right thing. It doesn't.

I think most people probably don't even use shared due to lacking Phobos support. E.g. http://d.puremagic.com/issues/show_bug.cgi?id=7036

Not even using the write functions worked on shared types until 2.059
(e.g. printing shared arrays).

'shared' has this wonderfully attractive name to it, but apparently it doesn't have much guarantees? E.g. Walter's comment here: http://d.puremagic.com/issues/show_bug.cgi?id=8077#c1

So +1 from me just because I have no idea what shared is supposed to
guarantee. I've just stubbornly used __gshared variables because
std.concurrency.send() doesn't accept mutable data. send() doesn't
work with shared either, so I have no clue.. :)

Fully agree.

Kind Regards
Benjamin Thaut

Alex Rønne Petersen:

> Something needs to be done about shared. I don't know what,

Maybe deprecate it and introduce something else that is rather different and based on thought-out theory?

Bye,
bearophile

On Sunday, 11 November 2012 at 19:28:30 UTC, Andrej Mitrovic wrote:
> On 11/11/12, Alex Rønne Petersen <alex@lycus.org> wrote:
>> And finally, the worst part of all of this? People writing code that
>> uses shared today are blindly assuming it actually does the right thing.
>> It doesn't.
>
> I think most people probably don't even use shared due to lacking
> Phobos support. E.g.
> http://d.puremagic.com/issues/show_bug.cgi?id=7036
>
> Not even using the write functions worked on shared types until 2.059
> (e.g. printing shared arrays).
>
> 'shared' has this wonderfully attractive name to it, but apparently it
> doesn't have much guarantees? E.g. Walter's comment here:
> http://d.puremagic.com/issues/show_bug.cgi?id=8077#c1
>
> So +1 from me just because I have no idea what shared is supposed to
> guarantee. I've just stubbornly used __gshared variables because
> std.concurrency.send() doesn't accept mutable data. send() doesn't
> work with shared either, so I have no clue.. :)

Fix support for shared(T) in std.variant, and you will have fixed send() as well.  Meanwhile, in common cases a simple wrapper struct suffices.
##################################################
module toy;
import std.concurrency, std.stdio;

struct SImpl {
    string s;
    int i;
}
alias shared( SImpl ) S;

struct Msg { S s; }
struct Quit {}

S global = S( "global", 999 );

void main () {
    auto child = spawn( &task );
    S s = S( "abc", 42 );
    child.send( Msg( s ) );
    child.send( Msg( global ) );
    child.send( Quit() );
}

void task () {
    bool sentinel = true;
    while ( sentinel ) {
        receive(
            ( Msg msg  ) { writeln( msg.s.s, " -- ", msg.s.i ); },
            ( Quit msg ) { sentinel = false; }
        );
    }
}
##################################################

grant@aesgard ~/Projects/D/foo/shared_test $ dmd toy && ./toy
abc -- 42
global -- 999

-- Chris Nicholson-Sauls

On Sunday, 11 November 2012 at 20:08:25 UTC, Benjamin Thaut wrote:
> Fully agree.

+1

On Sun, 11 Nov 2012 22:19:08 +0100, martin wrote:

> On Sunday, 11 November 2012 at 20:08:25 UTC, Benjamin Thaut wrote:
>> Fully agree.
> 
> +1

+1.

I find it so broken that I have to avoid using it in all but the most trivial situations.

Le 11/11/2012 23:36, Graham St Jack a écrit :
> On Sun, 11 Nov 2012 22:19:08 +0100, martin wrote:
>
>> On Sunday, 11 November 2012 at 20:08:25 UTC, Benjamin Thaut wrote:
>>> Fully agree.
>>
>> +1
>
> +1.
>
> I find it so broken that I have to avoid using it in all but the most
> trivial situations.

That isn't a bad thing in itself.

On Monday, November 12, 2012 01:17:06 deadalnix wrote:
> Le 11/11/2012 23:36, Graham St Jack a écrit :
> > On Sun, 11 Nov 2012 22:19:08 +0100, martin wrote:
> >> On Sunday, 11 November 2012 at 20:08:25 UTC, Benjamin Thaut wrote:
> >>> Fully agree.
> >> 
> >> +1
> > 
> > +1.
> > 
> > I find it so broken that I have to avoid using it in all but the most trivial situations.
> 
> That isn't a bad thing in itself.

I don' think that it's really intended that shared by 100% easy to use. You're _supposed_ to use it sparingly. But at this point, it borders on being utterly unusable.

We have a bit of a problem with the basic idea though in that you're not supposed to be using shared much, and it's supposed to be segregated such that having the shared equivalent of const (as in it works with both shared and non-shared) would pose a big problem (it's also probably untenable with memory barriers and the like), but if you _don't_ have something like that, you either can't use shared with much of anything, or you have to cast it away all over the place, which loses all of the memory barriers or whatnot. We have conflicting requirements which aren't being managed very well.

I don't know how protected shared really needs to be though. Anything involving shared should make heavy use of mutexes and synchronized and whatnot meaning that at least some of the protections that people want with shared are useless unless you're writing code which is being stupid and not using mutexes or whatnot. So, casting away shared might not actually be that big a deal so long as it's temporary to call a function (as opposed to stashing the variable away somewhere) and that call is protected by a mutex or other thread- protection mechanism.

At the moment, I think that the only way to make stuff work with both shared and unshared (aside from using lots of casts) is to make use of templates, and since most of druntime and Phobos isn't tested with shared, things like Unqual probably screw with that pretty thoroughly. It's at least conceivable though that stuff like std.algorithm could work with shared just fine.

I don't think that there's much question though that shared is the major chink in our armor with regards to thread-local by default. The basic idea is great, but the details still need some work.

- Jonathan M Davis

drop it in favour of :
http://forum.dlang.org/post/k7j1ta$2kv8$1@digitalmars.com


On Sunday, 11 November 2012 at 18:46:12 UTC, Alex Rønne Petersen wrote:
> Hi,
>
> It's starting to get outright embarrassing to talk to newcomers about D's concurrency support because the most fundamental part of it -- the shared type qualifier -- does not have well-defined semantics at all.
>
> I'm certainly not alone in being annoyed by this state of affairs: http://d.puremagic.com/issues/show_bug.cgi?id=8993
>
> I've posted rants about the state of shared before and, from the comments on those, it appears that what most people want shared to do is at least one (and usually multiple) of
>
> * make variables global (if appropriate in the context);
> * make the wrapped type completely separate from the unwrapped type;
> * make all operations be atomic;
> * make all operations result in memory barriers.
>
> At a glance, this looks fine. Exactly what you would want for shared types in a concurrent setting, right?
>
> Except, not really. I'll try to explain all of the unsolved problems with shared below...
>
> First of all, the fact that shared(T) is completely separate from T (i.e. no conversions allowed, except for primitive types) is a huge usability problem. In practice, it means that 99% of the standard library is unusable with shared types. Hell, even most of the runtime doesn't work with shared types. I don't know how to best solve this particular problem; I'm just pointing it out because anyone who tries to do anything non-trivial with shared will invariably run into this.
>
> Second, the idea of making shared insert atomic operations is an absolute fallacy. It only makes sense for primitive types for the most part, and even for those, what sizes are supported depends on the target architecture. A number of ideas have come up to solve this problem:
>
> * We make shared(T) not compile for certain Ts depending on the target architecture. I personally think this is a terrible idea because most code using shared will not be portable at all.
> * We require any architecture D targets to support atomic operations for a certain size S at the very least. This is fine for primitives up to 64 bits in size, but doesn't clear up the situation for larger types (real, complex types, cent/ucent, ...).
> * We make shared not insert atomic operations at all (thus making it kind of useless for anything but documentation).
> * (Possibly others I have forgotten; please let me know if this is the case.)
>
> I don't think any of these are particularly attractive, to be honest. If we do make shared insert atomic operations, we would also have to consider the memory ordering of those operations.
>
> Third, we have memory barriers. I strongly suspect that this is a misnomer in most cases where people have suggested this; it's generally not useful to have a compiler insert barriers because they are used to control ordering of load/store operations which is something the programmer will want to do explicitly. In any case, the compiler can't usefully figure out where to put barriers, so it would just result in really bad performance for no apparent gain.
>
> Fourth, there is implementation complexity. If shared is meant to insert specialized instructions, it will result in effectively two code paths for most code generation in any D compiler (read: maintenance nightmare).
>
> Fifth, it is completely unclear whether casting to and from shared is legal (but with a big fat "caution" sign like casting away const) or if it's undefined behavior. Making it undefined behavior would further increase the usability problem I described above.
>
> And finally, the worst part of all of this? People writing code that uses shared today are blindly assuming it actually does the right thing. It doesn't. Their code will break on any non-x86 platform. This is an absolutely horrifying situation now that ARM, MIPS, and PowerPC are starting to become viable targets for D.
>
> Something needs to be done about shared. I don't know what, but the current situation is -- and I'm really not exaggerating here -- laughable. I think we either need to just make it perfectly clear that shared is for documentation purposes and nothing else, or, figure out an alternative system to shared, because I don't see shared actually being useful for real world work no matter what we do with it.