Fri, 08 Jan 2010 23:12:38 +0000, dsimcha wrote:

> == Quote from Walter Bright (newshound1@digitalmars.com)'s article
>> Sure. Except that implicit parallelism is inherently unsound unless the
>> type system can support it.
>> I'll go further than that. Any language that supports implicit sharing
>> of mutable data is inherently unsound for multithreaded programming.
> 
> One thing that I think needs to be considered in the concurrency architecture is the case of performance-critical massive data parallelism.  In these cases concurrency is actually not very hard and a simple parallel foreach loop covers a lot of cases.  As far as safety, the amount of code being executed inside a parallel foreach loop is generally small enough that it's easy to reason about, and thus it's ok not to have any hard static guarantees and leave safety up to the programmer, as long as the programmer understands at least the basics of concurrency.
> 
> I would like to see a situation where OpenMP/ParallelFuture-style concurrency is still implementable in D without unreasonable performance or syntactic overhead after the new concurrency system is fully in place.

These systems also solve different kinds of problems. MMX/SSE/Altivec and now GPU hardware solves problems where you run a single operation on a large set of data simultaneously. The communication is really cheap because e.g. in SSE data is in "ordinary" registers which are the fastest memory units available.

Message passing system scales well if you have many computers that form a computing cluster. It also performs well when the computation can be split into distinct tasks which run in separate threads and communicate rarely. Unless messages are translated into native SIMD primitives in data parallel cases, message passing will be much slower when you do that kind of processing. Parallel loops don't scale that well on multiple computing nodes - the computation needs to be split into larger tasks to minimize communication delays since nodes simply can't share the same thread state.

dsimcha wrote:
> == Quote from Walter Bright (newshound1@digitalmars.com)'s article
>> Sure. Except that implicit parallelism is inherently unsound unless the
>> type system can support it.
>> I'll go further than that. Any language that supports implicit sharing
>> of mutable data is inherently unsound for multithreaded programming.
> 
> One thing that I think needs to be considered in the concurrency architecture is
> the case of performance-critical massive data parallelism.  In these cases
> concurrency is actually not very hard and a simple parallel foreach loop covers a
> lot of cases.  As far as safety, the amount of code being executed inside a
> parallel foreach loop is generally small enough that it's easy to reason about,
> and thus it's ok not to have any hard static guarantees and leave safety up to the
> programmer, as long as the programmer understands at least the basics of concurrency.

It's hard to reason about even small examples. Consider the double checked locking bug. I even thought I understood it, until Bartosz discovered I'd written one into Phobos.

As for understanding the basics of concurrency, I defy anyone to read the Intel CPU manuals and discover how memory read/write barriers work on Intel chips and how and when they should be used.

With C++, everyone using memory barriers has to figure it out, and the consequences of getting it wrong are very, very, very obscure bugs. With D, only the compiler writer has to get it right.


> I would like to see a situation where OpenMP/ParallelFuture-style concurrency is
> still implementable in D without unreasonable performance or syntactic overhead
> after the new concurrency system is fully in place.

There's no reason it cannot be.

At the worst, you can cast away "shared" and cowboy it exactly as you would in C++.

There is no circumstance where you are worse off with D than with C++, and quite a lot of cases where you're much better off.

== Quote from Walter Bright (newshound1@digitalmars.com)'s article
> dsimcha wrote:
> > == Quote from Walter Bright (newshound1@digitalmars.com)'s article
> >> Sure. Except that implicit parallelism is inherently unsound unless the
> >> type system can support it.
> >> I'll go further than that. Any language that supports implicit sharing
> >> of mutable data is inherently unsound for multithreaded programming.
> >
> > One thing that I think needs to be considered in the concurrency architecture is the case of performance-critical massive data parallelism.  In these cases concurrency is actually not very hard and a simple parallel foreach loop covers a lot of cases.  As far as safety, the amount of code being executed inside a parallel foreach loop is generally small enough that it's easy to reason about, and thus it's ok not to have any hard static guarantees and leave safety up to the programmer, as long as the programmer understands at least the basics of
concurrency.
> It's hard to reason about even small examples. Consider the double
> checked locking bug. I even thought I understood it, until Bartosz
> discovered I'd written one into Phobos.
> As for understanding the basics of concurrency, I defy anyone to read
> the Intel CPU manuals and discover how memory read/write barriers work
> on Intel chips and how and when they should be used.
> With C++, everyone using memory barriers has to figure it out, and the
> consequences of getting it wrong are very, very, very obscure bugs. With
> D, only the compiler writer has to get it right.
> > I would like to see a situation where OpenMP/ParallelFuture-style concurrency is still implementable in D without unreasonable performance or syntactic overhead after the new concurrency system is fully in place.
> There's no reason it cannot be.
> At the worst, you can cast away "shared" and cowboy it exactly as you
> would in C++.
> There is no circumstance where you are worse off with D than with C++,
> and quite a lot of cases where you're much better off.

Ok, one thing that leads to a lot of misunderstanding and confusion is that there's (as far as I can tell) absolutely no documentation of what shared does/will do or what its semantics are/will be, other than a few newsgroup posts from years ago,which were tentative, and the "migrating to shared" article, which is mostly about thread-local storage.  Also, as far as I can tell, shared is not fully implemented yet, because core.thread doesn't require a shared delegate.  Is this correct?

Here are some questions, and this is just what I've been able to think of off the top of my head in a few minutes:

What guarantees is shared supposed to provide?
What does shared have to do with synchronization?
What does shared have to do with memory barriers?
What are the semantics of casting FROM unshared TO shared?
What are the semantics of casting FROM shared TO unshared?
Why doesn't shared appear anywhere in core.thread?

dsimcha wrote:
> Ok, one thing that leads to a lot of misunderstanding and confusion is that
> there's (as far as I can tell) absolutely no documentation of what shared
> does/will do or what its semantics are/will be, other than a few newsgroup posts
> from years ago,which were tentative, and the "migrating to shared" article, which
> is mostly about thread-local storage.  Also, as far as I can tell, shared is not
> fully implemented yet, because core.thread doesn't require a shared delegate.  Is
> this correct?

Yes.

> Here are some questions, and this is just what I've been able to think of off the
> top of my head in a few minutes:
> 
> What guarantees is shared supposed to provide?

Shared means that multiple threads can access the data. The guarantee is that if it is not shared, and not immutable, that only the current thread can see it.

> What does shared have to do with synchronization?

Only shared data can be synchronized. It makes no sense to synchronize thread local data.

> What does shared have to do with memory barriers?

Reading/writing shared data emits memory barriers to ensure sequential consistency (not implemented).

> What are the semantics of casting FROM unshared TO shared?

You'd better be sure there are no other unshared references to that same data.

> What are the semantics of casting FROM shared TO unshared?

You'd better be sure there are no other shared references to that same data.

> Why doesn't shared appear anywhere in core.thread?

Because nobody has spent the effort to do it.

dsimcha Wrote:
>
> Why doesn't shared appear anywhere in core.thread?

Until the semantics of shared are finalized and the critical bugs are worked out I don't want to integrate shared with core.thread--it's the centerpoint of too much stuff (GC, etc).  But general purpose concurrency should never touch core.thread anyway.  There will be a higher-level interface available that's safer and easier to use.  One could also argue that core.thread should accept unshared delegates, since it's possible to do so safely, you just don't have any language support to help.  The design philosophy with the modules in core is quite different from std.

Walter Bright wrote:
> dsimcha wrote:
>> Ok, one thing that leads to a lot of misunderstanding and confusion is that
>> there's (as far as I can tell) absolutely no documentation of what shared
>> does/will do or what its semantics are/will be, other than a few newsgroup posts
>> from years ago,which were tentative, and the "migrating to shared" article, which
>> is mostly about thread-local storage.  Also, as far as I can tell, shared is not
>> fully implemented yet, because core.thread doesn't require a shared delegate.  Is
>> this correct?
> 
> Yes.
> 
>> Here are some questions, and this is just what I've been able to think of off the
>> top of my head in a few minutes:
>>
>> What guarantees is shared supposed to provide?
> 
> Shared means that multiple threads can access the data. The guarantee is that if it is not shared, and not immutable, that only the current thread can see it.
> 
>> What does shared have to do with synchronization?
> 
> Only shared data can be synchronized. It makes no sense to synchronize thread local data.
> 
>> What does shared have to do with memory barriers?
> 
> Reading/writing shared data emits memory barriers to ensure sequential consistency (not implemented).
> 
>> What are the semantics of casting FROM unshared TO shared?
> 
> You'd better be sure there are no other unshared references to that same data.
> 
>> What are the semantics of casting FROM shared TO unshared?
> 
> You'd better be sure there are no other shared references to that same data.
> 
>> Why doesn't shared appear anywhere in core.thread?
> 
> Because nobody has spent the effort to do it.

Thanks Walter for doing this. These questions and their answers could start an FAQ page on D's website.

Andrei

On 2010-01-08 03:50:26 -0500, Walter Bright <newshound1@digitalmars.com> said:

> Michel Fortin wrote:
>> Also keep in mind that we don't really need a shared vision among everyone. What's needed is someone who takes the decisions. Discussion is only needed to help that person take the right decisions. Although consensus among all members certainly boosts the decider self-confidence, it is not required, and not necessarily desirable either. A consensus among only a few key people is all that is needed, and this has little to do with who is allowed to raise issues and propose solutions.
> 
> The real problem with a concurrency model is that very few programmers understand the issues.
> [...]
> 
> Since then I have tried to master this topic, but I don't have much experience writing complex multithreaded code. So what we need are people who are experienced with MT code who can evaluate the design to see if we missed the boat or not. I'd rather not shoot at the moon yet wind up orbiting some asteroid.

I'm not sure what you're replying to in the above message. I was simply saying that to accomplish progress in something, the only necessary condition is to have an understanding among those who will actually implement that thing. Other opinions can be very useful, they can also be irrelevant, but you don't absolutely need a consensus among every participant in the discussion for something to advance.

You're making a case for making "shared" help programmers because most don't grasp all the implications of shared memory between processors. That's good but I don't really see how it relates.

By the way, I have seen my lot of multithreaded code in C++, sometime very well written, sometime just wrong, and sometime correct but obscure enough that takes help from the original author to understand. I've written code that probably enters in all these categories, and debugged my share of hard to track concurrency issues too. Were I was working previously, the most time-consuming bugs were always concurrency issues, generally race conditions happening every few hours/days or in certain rare conditions but not always, causing either deadlocks or other problematic behaviors. Fortunately, we also had failsafes to stop things in case the software would crash as it had safety implications.

Although I'm no longer working there and dealing much less with concurrency issues now, I'm still interested in the subject.

-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/

Andrei Alexandrescu wrote:

> Thanks Walter for doing this. These questions and their answers could start an FAQ page on D's website.
>
> Andrei

In the mean time: http://www.prowiki.org/wiki4d/wiki.cgi?DevelopmentWithD/FAQ/Shared