== Quote from Sean Kelly (sean@invisibleduck.org)'s article
> I really need to fix this.  It's a pain though, for the reasons related to the ones you mention in "5. Shared".  A tid, for example, fronts a message queue object that has some shared interface elements and some unshared interface elements.  The shared portion, rather than labeling functions as synchronized, uses synchronized internally to make the mutex use as fine-grained as possible.  And the logically unshared portion only synchronizes when accessing the shared data in the object. So to make Tid work with shared I would basically have to label the shared methods as shared and re-evaluate the ASM output once the compiler inserts memory barriers, and cast away shared when accessing the message queue from within its owner thread.  What gets me about all this is that rather than helping me, the type system is working against me.  I love shared as far as its use for globals is concerned, and for concrete variables, but not so much for classes.

Yea, shared really was a blunder at least in anything like its current form.  I think it's time we just admit it and do our best to mitigate it or find a way to massively overhaul it (though I'm skeptical that it can be overhauled successfully, especially with the constraints on backwards compatibility).

The default thread-local/explicit global was a great idea, as was providing a share-nothing-except-immutable message passing-based concurrency module in Phobos for people who want safe, simple, coarse-grained concurrency.  Message passing is safe and good for a lot of stuff, but not everything.

Once you're trying to use a threading paradigm where you need shared mutable
state, though, it can't be safe and it's a waste of time for the language to try.
 Even if you get rid of low-level data races, you still need to worry about
high-level invariants, so you've only won half the battle.  Furthermore, the
shared type constructor cripples shared-state multithreading so much that it's
almost useless unless you do some casting, defeating its purpose.  The way I see
this evolving is that almost all multithreaded code in D will either:

1.  Avoid sharing and just use some combination of straight message passing and immutable data.

2.  Bypass shared with casts, use of core.thread, std.parallelism, etc. and just do threading the old-fashioned way. (Though shared-state multithreading can be made less dangerous by encapsulating high-level paradigms.  In this respect std.parallelism represents a middle ground between the completely safe std.concurrency and the completely flexible core.thread.)

Ironically, I don't see this as such a bad outcome, except for the wasted "shared" keyword and dead trees describing it.  D is a systems language and there needs to be ways to do dangerous, unchecked multithreading.  It's great to provide a safe but limited way to write concurrent programs (such as share-nothing message passing), it's a no-brainer to prohibit the dangerous ways in SafeD, and it's fine to require some explicitness when using the dangerous ways (such as importing a different module).  However, fighting the type system every inch of the way while paying lip service to playing nice with shared is not an acceptable solution.  If you need shared state then you're almost guaranteed to need to cast away shared all over the place, and if you need to do so then you may as well bypass it entirely because it's just getting in the way and not providing any safety.  This is why I've been so adamant about keeping core.thread and std.parallelism the way they are unless shared massively improves in ways that I'm very skeptical are even possible.

On Aug 30, 2011, at 1:41 PM, dsimcha wrote:

> == Quote from Sean Kelly (sean@invisibleduck.org)'s article
>> I really need to fix this.  It's a pain though, for the reasons related to the ones you mention in "5. Shared".  A tid, for example, fronts a message queue object that has some shared interface elements and some unshared interface elements.  The shared portion, rather than labeling functions as synchronized, uses synchronized internally to make the mutex use as fine-grained as possible.  And the logically unshared portion only synchronizes when accessing the shared data in the object. So to make Tid work with shared I would basically have to label the shared methods as shared and re-evaluate the ASM output once the compiler inserts memory barriers, and cast away shared when accessing the message queue from within its owner thread.  What gets me about all this is that rather than helping me, the type system is working against me.  I love shared as far as its use for globals is concerned, and for concrete variables, but not so much for classes.
> 
> Yea, shared really was a blunder at least in anything like its current form.  I think it's time we just admit it and do our best to mitigate it or find a way to massively overhaul it (though I'm skeptical that it can be overhauled successfully, especially with the constraints on backwards compatibility).
> 
> The default thread-local/explicit global was a great idea, as was providing a share-nothing-except-immutable message passing-based concurrency module in Phobos for people who want safe, simple, coarse-grained concurrency.  Message passing is safe and good for a lot of stuff, but not everything.
> 
> Once you're trying to use a threading paradigm where you need shared mutable state, though, it can't be safe and it's a waste of time for the language to try.

I think the idea behind shared as it applies to classes is that the desired application won't have many shared objects, and those that are shared should typically be fairly simple things like containers of objects that don't have external references.  The typical "web of shared objects" as per Java is an invitation to disaster, and so the model rightly discourages that.

Regarding classes, I think one fundamental problem is that shared is transitive.  Simply putting memory barriers between accesses to shared data doesn't mean that the algorithm itself will be correct if executed concurrently.  Transitively applying shared to member data and allowing lock-free operations on this data and having the compiler even insert memory barriers suggests to the neophyte user (IMO) that this is a safe and acceptable means for making a class multithreaded when nothing could be further from the truth.  Another issue with transitivity is how this affects C API calls, as the recent flurry of Windows patches to Phobos can attest.


> Even if you get rid of low-level data races, you still need to worry about high-level invariants, so you've only won half the battle.  Furthermore, the shared type constructor cripples shared-state multithreading so much that it's almost useless unless you do some casting, defeating its purpose.  The way I see this evolving is that almost all multithreaded code in D will either:
> 
> 1.  Avoid sharing and just use some combination of straight message passing and immutable data.
> 
> 2.  Bypass shared with casts, use of core.thread, std.parallelism, etc. and just do threading the old-fashioned way. (Though shared-state multithreading can be made less dangerous by encapsulating high-level paradigms.  In this respect std.parallelism represents a middle ground between the completely safe std.concurrency and the completely flexible core.thread.)

Regarding #1, I do think there's a case to be made for having mutable shared data, but that data can't be terribly complex.  I do very much dislike having to cast away shared though, because it makes for a very awkward API design.  For example, say I want to use synchronized in a fine-grained manner.  I have to do something like this:

class MyClass {
    shared void doSomething() {
        cast(Unshared!(MyClass)).doSomething_();
    }
    private void doSomething_() {
       …
       synchronized(this) { … }
       …
    }


> Ironically, I don't see this as such a bad outcome, except for the wasted "shared" keyword and dead trees describing it.  D is a systems language and there needs to be ways to do dangerous, unchecked multithreading.  It's great to provide a safe but limited way to write concurrent programs (such as share-nothing message passing), it's a no-brainer to prohibit the dangerous ways in SafeD, and it's fine to require some explicitness when using the dangerous ways (such as importing a different module).  However, fighting the type system every inch of the way while paying lip service to playing nice with shared is not an acceptable solution.  If you need shared state then you're almost guaranteed to need to cast away shared all over the place, and if you need to do so then you may as well bypass it entirely because it's just getting in the way and not providing any safety.  This is why I've been so adamant about keeping core.thread and std.parallelism the way they are unless shared massively improves in ways that I'm very skeptical are even possible.

And this is why I haven't made much effort to change core.thread.  I've actually tried twice so far and gave up each time after seeing the cascade of changes necessary.

I'd like to say that the 'shared' attribute on member functions should just mean "make this member visible through a shared reference" and do away with the memory barriers idea entirely, except that D really does need some form of atomics.  I'm really not sure what the solution is here.

On 8/30/2011 8:01 PM, Sean Kelly wrote:

> I'd like to say that the 'shared' attribute on member functions should just mean "make this member visible through a shared reference" and do away with the memory barriers idea entirely, except that D really does need some form of atomics.  I'm really not sure what the solution is here.

What's wrong with core.atomic?

On 8/30/2011 8:01 PM, Sean Kelly wrote:
> I'd like to say that the 'shared' attribute on member functions should just mean "make this member visible through a shared reference"

So the idea would just be that shared is a programmer-verified seal of approval that "yes, this is thread-safe without the user of the object taking any additional precautions"?

On 2011-08-31 00:01:09 +0000, Sean Kelly <sean@invisibleduck.org> said:

> I'd like to say that the 'shared' attribute on member functions should
> just mean "make this member visible through a shared reference" and do
> away with the memory barriers idea entirely, except that D really does
> need some form of atomics.  I'm really not sure what the solution is
> here.

I think D needs some kind of knowledge about memory regions into its type system. The challenge is to not add too much complexity.


-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/

Am 29.08.2011 23:12, schrieb Walter Bright:
> On 8/27/2011 10:14 AM, Benjamin Thaut wrote:
>> Comments and criticism welcome
>
>
> This guy wants to contact you:
>
> http://www.reddit.com/r/programming/comments/jwkvx/suggestions_for_the_d_20_programming_language/c2g2wos
>

Thanks for the heads up

-- 
Kind Regards
Benjamin Thaut

Sean Kelly Wrote:

> The shared portion, rather than labeling functions as synchronized, uses synchronized internally to make the mutex use as fine-grained as possible.  And the logically unshared portion only synchronizes when accessing the shared data in the object.

What's the difference? Both access shared data and synchronize. If the data accessed is unshared, there should be no problem to do it from shared method - there will be no barriers because the data is unshared, as I understand it.

I meant atomic types, though I could probably be convinced otherwise.

Sent from my iPhone

On Aug 30, 2011, at 5:25 PM, dsimcha <dsimcha@yahoo.com> wrote:

> On 8/30/2011 8:01 PM, Sean Kelly wrote:
> 
>> I'd like to say that the 'shared' attribute on member functions should just mean "make this member visible through a shared reference" and do away with the memory barriers idea entirely, except that D really does need some form of atomics.  I'm really not sure what the solution is here.
> 
> What's wrong with core.atomic?

The effect on storage for statics is fantastic. I wasn't suggesting changing that, if that's what you meant.

Sent from my iPhone

On Aug 30, 2011, at 5:42 PM, Michel Fortin <michel.fortin@michelf.com> wrote:

> On 2011-08-31 00:01:09 +0000, Sean Kelly <sean@invisibleduck.org> said:
> 
>> I'd like to say that the 'shared' attribute on member functions should just mean "make this member visible through a shared reference" and do away with the memory barriers idea entirely, except that D really does need some form of atomics.  I'm really not sure what the solution is here.
> 
> I think D needs some kind of knowledge about memory regions into its type system. The challenge is to not add too much complexity.
> 
> 
> -- 
> Michel Fortin
> michel.fortin@michelf.com
> http://michelf.com/
> 

The unshared methods are only ever meant to be called by the queue owner. They access some member data without synchronization to make the queue efficient for receive calls. I could make these methods shared anyway, but if they were called concurrently by accident it would mean data corruption.

Sent from my iPhone

On Aug 31, 2011, at 12:37 AM, Kagamin <spam@here.lot> wrote:

> Sean Kelly Wrote:
> 
>> The shared portion, rather than labeling functions as synchronized, uses synchronized internally to make the mutex use as fine-grained as possible.  And the logically unshared portion only synchronizes when accessing the shared data in the object.
> 
> What's the difference? Both access shared data and synchronize. If the data accessed is unshared, there should be no problem to do it from shared method - there will be no barriers because the data is unshared, as I understand it.