freeagle wrote:
> Why do people think there is a need for another language/paradigm to solve concurrent problem? OSes deal with parallelism for decades, without special purpose languages. Just plain C, C++. Just check Task manager in windows and you'll notice there's about 100+ threads running.
> If microsoft can manage it with current languages, why we cant?
> 
> freeagle

That's a good point... Services and daemons have been running on multi-cpu machines for years, and AFAICT even multi-core machines prior to Intel Core 2 didn't require "a whole new paradigm" [or even a whole new compiler or new libraries].

For language support, Java, its runtime and std. lib. for example were designed with concurrency in mind and I'm not so sure much more can reasonably be done with an imperative language (although I'm sure some improvements could be made).

How will the new Intel chip architecture really drastically change that? I recently read an article that many server applications are already "close to ideally suited" to the new CPU architectures, and "should immediately benefit" from them.

(If they can "immediately benefit" from the multi-core designs, then it's not a leap to suppose that these "old" techniques must still hold a good deal of merit. Why re-invent the wheel?).

Sure, there are some areas like "lock-free hastables", theading libs. and such that will need to be done differently to get the _most_ benefit from the new architectures, but I don't think it will or need to go much beyond that (I'm not so sure that other more complex things like Garbage Collection will need to be re-developed to take advantage of multi-core).

What I think will need to change for the most part will be how _some_ "fat client" applications are developed, but that is becoming less relevant in this era of "thin-client" computing. IIS, Apache, Oracle, SQL Server and the rest take care of most of the concurrent operation worries for us <g> OTOH, maybe multi-core becoming available on a typical client will create a resurgence of demand for "fat-client".

- Dave

Henrik wrote:
> Todays rant on Slashdot is about parallel programming and why the support for multiple cores in programs is only rarely seen. There are a lot of different opinions on why we haven’t seen a veritable rush to adopt parallelized programming strategies, some which include:
> 
> * Multiple cores haven't been available/affordable all that long, programmers just need some time to catch up.
> * Parallel programming is hard to do (as we lack the proper programming tools for it). We need new concepts, new tools, or simply a new generation of programming languages created to handle parallelization from start.
> * Parallel programming is hard to do (as we tend to think in straight lines, lacking the proper cognitive faculties to parallelize problem solving). We must accept that this is an inherently difficult thing for us, and that there never will be an easy solution.
> * We have both the programming tools needed and the cognitive capacity to deal with them, only the stupidity of the current crop of programmers or their inability to adapt stand in the way. Wait a generation and the situation will have sorted itself out.
> 
> I know concurrent programming has been a frequent topic in the D community forums, so I would be interested to hear the community’s opinions on this. What will the future of parallel programming look like? Are new concepts and tools that support parallel programming needed, or just a new way of thinking? Will the “old school programming languages fade away, as some seem to suggest, to be replaced by HOFL:s (Highly Optimized Functional Languages)? Where will/should D be in all this? Is it a doomed language if it does incorporate an efficient way of dealing with this (natively)?
> 
> 
> Link to TFA: http://developers.slashdot.org/developers/07/05/29/0058246.shtml
> 
> 
> /// Henrik
> 

I think if D were to grow parallel legs, I think it would be a great incentive for people to make the switch.

-Joel

Henrik Wrote:

> Todays rant on Slashdot is about parallel programming and why the
... [snip]

At work, I'm using SEDA:

http://www.eecs.harvard.edu/~mdw/papers/mdw-phdthesis.pdf

Although it's designed primarily for internet services (and indeed, I'm crafting an internet service...), I'm using it for a lot more than just the server portions of the program, and I plan to use it in future (non-internet-based) solutions.

The general idea behind something like this is "fire and forget". There is a set of "stages", each with a queue of events, and a thread pool varying in size depending on load (all managed behind the scenes). The creator of a stage needs only to receive an event and process it, possibly (usually), pushing other events onto other stages, where they will be executed when there's time. Stages are highly modular, and tend to serve only one, or a small group of, functions, but each stage is managed behind the scenes with monitoring tools that increase and decrease thread count respective to thread load.

The advantage of such a system is it allows the designer to think in a very "single-threaded" mindset. You deal with a single event, and when you're done processing it, you let someone else (or a few other people) deal with the results. It also encourages encapsulation and modularity.

The disadvantage? It's not suited for all types of software. It's ideal for server solutions, and I could see its use in various GUI apps, but it might be hard to force an event-driven model onto something like a game.

Combined with something like the futures paradigm, though, I can see this being very helpful for allowing multi-threaded code to be written like single-threaded code.

Now only to port it to D...

Robert Fraser wrote:
> Henrik Wrote:
> 
>> Todays rant on Slashdot is about parallel programming and why the
> ... [snip]
> 
> At work, I'm using SEDA:
> 
> http://www.eecs.harvard.edu/~mdw/papers/mdw-phdthesis.pdf

Wow.

Anyone doubting if this makes a difference should compare the graphs on pages 18 and 25.

> 
> Although it's designed primarily for internet services (and indeed, I'm crafting an internet service...), I'm using
> it for a lot more than just the server portions of the program, and I plan to use it in future (non-internet-based)
> solutions.
> 
> The general idea behind something like this is "fire and forget". There is a set of "stages", each with a queue of
> events, and a thread pool varying in size depending on load (all managed behind the scenes). The creator of a stage
> needs only to receive an event and process it, possibly (usually), pushing other events onto other stages, where they
> will be executed when there's time. Stages are highly modular, and tend to serve only one, or a small group of,
> functions, but each stage is managed behind the scenes with monitoring tools that increase and decrease thread count
> respective to thread load.
> 
> The advantage of such a system is it allows the designer to think in a very "single-threaded" mindset. You deal with
> a single event, and when you're done processing it, you let someone else (or a few other people) deal with the
> results. It also encourages encapsulation and modularity.
> 
> The disadvantage? It's not suited for all types of software. It's ideal for server solutions, and I could see its use
> in various GUI apps, but it might be hard to force an event-driven model onto something like a game.

Actually, just about everything in most modern 3d games is event driven, except for the renderer. FWIW, the renderer simply redraws the screen on a zen timer, based on what's sitting in the render queue, so it can easily be run in parallel with the event pump.  The event pump, in turn, modifies the render queue.

The only difference between a game and a typical GUI app is that even modest event latency can be a showstopper.  The renderer *must* run on time (or else you drop frames), I/O events must be handled quickly (sluggish handling), game/entity events must be fast, and render queue contention must be kept very low.

> 
> Combined with something like the futures paradigm, though, I can see this being very helpful for allowing
> multi-threaded code to be written like single-threaded code.
> 
> Now only to port it to D...


-- 
- EricAnderton at yahoo

Daniel Keep Wrote:
> freeagle wrote:
> > Why do people think there is a need for another language/paradigm to solve concurrent problem? OSes deal with parallelism for decades, without special purpose languages. Just plain C, C++. Just check Task manager in windows and you'll notice there's about 100+ threads running. If microsoft can manage it with current languages, why we cant?
> > 
> > freeagle
> 
> We can; it's just hard as hell and thoroughly unenjoyable.  Like I said before: I can and have written multithreaded code, but it's so utterly painful that I avoid it wherever possible.

I must be strange then because after 5+ years of multithreaded programming it's the sort I prefer to do.  Each to their own I guess.

I think perhaps it's something that can be learnt, but it takes a bit of time, similar in fact to learning to program in the first place.  I enjoy the challenge of it and I think once you understand the fundamental problems/rules/practices with multithreaded development it becomes almost easy, almost.

> It's like trying to wash a car with a toothbrush and one of those giant novelty foam hands.  Yeah, you could do it, but wouldn't it be really nice if someone would go and invent the sponge and wash-cloth?

I think with some higher level constructs it becomes easier.  For example one of the main problems you face are deadlocks.  A common cause of a deadlocks is:

class A {
  void foo() {
    synchronize(this) { ... }
  }
}

void main()
{
  A a = new A();
  synchronize(a) {
    a.foo();
  }
}

now, I haven't tested the above in D but in Java and C# the 'lock' statement causes a deadlock above because you cannot lock the same object twice, even from the same thread.

In my previous job we used a mutex object that allowed the same thread to lock the same object any number of times, counting the number of lock calls and requiring an equal number of unlock calls.  This idea makes life a lot easier.  No more deadlocks of this type.

That just leaves the deadlock you get when you say:

synchronize(a) { synchronize(b) { .. } }

and in another thread:

synchronize(b) { synchronize(a) { .. } }

Given the right (or rather wrong) timing this can result in a dealock of both threads.  This situation is less common simply because it's less common for 2 blocks of code in 2 different threads to need 2 or more mutexes _at the same time_.  Or, at least, that is my experience.

In my previous job we went so far as to intentionally terminate the process if a deadlock was detected and could then give the exact file:line of the last lock request making it fairly trivial to debug.  We could do this because our server process had a 2nd process watching over it, and restarting it at a moments notice.

It's things like these which make multithreaded programming much easier for developers to get their head around.

I wonder what D's synchronized statement does?  Does it allow multiple locks from the same thread?  It should, especially given that it is impossible to forget the unlock call (that in itself is a great boon to multithreaded development).

Another good idea is to provide a ThreadPool construct, do we have one of those floating around (pun intended).  The idea being that when you need a thread you ask the pool for one and it supplies it, then once you're done with it you release it back into the pool for the next piece of code to pick it up and run with it (get it, run with it, like a ball... well I thought it was funny).

Just a few ideas.  Like several people have posted, the next great idea is probably waiting to be thought up!  but, in the meantime we can make life a little easier, one step at a time.

Regan Heath

Some ideas/terms for those that are interested:

http://en.wikipedia.org/wiki/Lock_%28computer_science%29 http://en.wikipedia.org/wiki/Mutual_exclusion http://en.wikipedia.org/wiki/Critical_section http://en.wikipedia.org/wiki/Semaphore_%28programming%29 http://en.wikipedia.org/wiki/Spinlock http://en.wikipedia.org/wiki/Seqlock

as you can see there are many ways to implement the humble 'synchronized' statement.  I wonder which D uses?

Regan

Regan Heath wrote:
> Daniel Keep Wrote:
>>>
>> We can; it's just hard as hell and thoroughly unenjoyable.  Like I said
>> before: I can and have written multithreaded code, but it's so utterly
>> painful that I avoid it wherever possible.
> 
> I must be strange then because after 5+ years of multithreaded programming it's the sort I prefer to do.  Each to their own I guess.

Same here.  That said, I still believe this is something the user shouldn't generally have to think about.  Let's say you want to sort a large array.  Is it better to do so using a single thread or multiple threads?  What if the app must adapt to use whatever resources are available, be that 1 CPU or 16 CPUs?  We are quickly heading towards an area where few competent multithreaded programmers even have experience.    It's not terribly difficult to target 2-4 CPUs because the number is sufficiently small that multi-threaded programs still look much like single-threaded programs.  But what if the target machine has 16 CPUs? 32?  The more parallel machines get the worse explicit multi-threading fits.  Eventually, I will want the compiler/VM to figure most of it out for me, or at least let me explicitly designate functions as atomic, etc.  Cilk is a decent example of how a C-like language could be adapted
 for multi-threading, but I still consider it an example of yesterday's solution, not tomorrow's.

> now, I haven't tested the above in D but in Java and C# the 'lock' statement causes a deadlock above because you cannot lock the same object twice, even from the same thread.

Locks are recursive in D (thank goodness).  But that only solves on eof the two problems you mention.

> In my previous job we used a mutex object that allowed the same thread to lock the same object any number of times, counting the number of lock calls and requiring an equal number of unlock calls.  This idea makes life a lot easier.  No more deadlocks of this type.

Yup, this is how D works.

> That just leaves the deadlock you get when you say:
> 
> synchronize(a) { synchronize(b) { .. } }
> 
> and in another thread:
> 
> synchronize(b) { synchronize(a) { .. } }
> 
> Given the right (or rather wrong) timing this can result in a dealock of both threads.  This situation is less common simply because it's less common for 2 blocks of code in 2 different threads to need 2 or more mutexes _at the same time_.  Or, at least, that is my experience.

This is why some people (like Herb Sutter) say that object-oriented programming is inherently incompatible with explicit multi-threading. Any call into unknown code risks deadlock, and the whole point of OOP is generalizing problems in a manner that requires calling into unknown code.  It's no wonder that message passing (as with CSP) seems to be gaining traction.

> I wonder what D's synchronized statement does?  Does it allow multiple locks from the same thread?  It should, especially given that it is impossible to forget the unlock call (that in itself is a great boon to multithreaded development).

Yes it does.

> Another good idea is to provide a ThreadPool construct, do we have one of those floating around (pun intended).  The idea being that when you need a thread you ask the pool for one and it supplies it, then once you're done with it you release it back into the pool for the next piece of code to pick it up and run with it (get it, run with it, like a ball... well I thought it was funny).

Tango will almost certainly have one prior to its 1.0 release.  It already has a ThreadGroup object, but this is more for simply grouping multiple threads than it is for providing a general means for performing async. tasks.  I'm not yet sure just how extensive multi-threading support will be added to Tango by 1.0, but definitely more than it has now.


Sean

Jeff Nowakowski wrote:
> freeagle wrote:
>> Why do people think there is a need for another language/paradigm to solve concurrent problem? OSes deal with parallelism for decades, without special purpose languages. Just plain C, C++. Just check Task manager in windows and you'll notice there's about 100+ threads running.
> 
> Why limit yourself to hundreds of threads when you can have thousands?

Because context switching is expensive.  Running thousands of threads on a system with only a few CPUs may use more time simply switching between threads than it does executing the thread code.


Sean

Dave wrote:
> 
> How will the new Intel chip architecture really drastically change that? I recently read an article that many server applications are already "close to ideally suited" to the new CPU architectures, and "should immediately benefit" from them.

Most of them are.  I think the problem will be more with user apps, particularly games.

> (If they can "immediately benefit" from the multi-core designs, then it's not a leap to suppose that these "old" techniques must still hold a good deal of merit. Why re-invent the wheel?).

Because the traditional means of multi-threading will not scale well to systems with a large number of CPUs (in my opinion).  By large I mean at least 16, which people at Intel have said we'll be using within a few years.  That doesn't give software developers much lead time to figure out how to easily use all that hardware.

> What I think will need to change for the most part will be how _some_ "fat client" applications are developed, but that is becoming less relevant in this era of "thin-client" computing. IIS, Apache, Oracle, SQL Server and the rest take care of most of the concurrent operation worries for us <g> OTOH, maybe multi-core becoming available on a typical client will create a resurgence of demand for "fat-client".

How many "thin client" applications do you use?  I don't use any, unless you count web forums.  If the "thin client" idea invades the desktop I suspect it will be as common for desktops to be running both the client and the server as it will to run only the client with a remote server. And that still leaves out games, which have been driving personal computer development for almost 20 years.  I suppose that means I think "fat clients" are the more likely scenario.


Sean

Sean Kelly Wrote:
> Regan Heath wrote:
> > Daniel Keep Wrote:
>  >>>
> >> We can; it's just hard as hell and thoroughly unenjoyable.  Like I said before: I can and have written multithreaded code, but it's so utterly painful that I avoid it wherever possible.
> > 
> > I must be strange then because after 5+ years of multithreaded programming it's the sort I prefer to do.  Each to their own I guess.
> 
> Same here.  That said, I still believe this is something the user
> shouldn't generally have to think about.  Let's say you want to sort a
> large array.  Is it better to do so using a single thread or multiple
> threads?  What if the app must adapt to use whatever resources are
> available, be that 1 CPU or 16 CPUs?  We are quickly heading towards an
> area where few competent multithreaded programmers even have experience.
>     It's not terribly difficult to target 2-4 CPUs because the number is
> sufficiently small that multi-threaded programs still look much like
> single-threaded programs.  But what if the target machine has 16 CPUs?
> 32?  The more parallel machines get the worse explicit multi-threading
> fits.  Eventually, I will want the compiler/VM to figure most of it out
> for me, or at least let me explicitly designate functions as atomic,
> etc.  Cilk is a decent example of how a C-like language could be adapted
>   for multi-threading, but I still consider it an example of yesterday's
> solution, not tomorrow's.

I can see your point... it's definately something to think about.

> > Given the right (or rather wrong) timing this can result in a dealock of both threads.  This situation is less common simply because it's less common for 2 blocks of code in 2 different threads to need 2 or more mutexes _at the same time_.  Or, at least, that is my experience.
> 
> This is why some people (like Herb Sutter) say that object-oriented programming is inherently incompatible with explicit multi-threading. Any call into unknown code risks deadlock, and the whole point of OOP is generalizing problems in a manner that requires calling into unknown code.  It's no wonder that message passing (as with CSP) seems to be gaining traction.

I need to do some reading about CSP and message passing.  I haven't poked my head outside my comfy shell for a good while now.

In the case I mention above you can at least solve it by giving each mutex an id, or priority.  Upon aquisition you ensure that no other mutex of lower priority is currently held, if it is you release both and re-aquire in the correct order (high to low or low to high whichever you decide, all that matters is that there is an order defined and adhered to in all cases).

Another higher level construct to consider for tango perhaps?

> > I wonder what D's synchronized statement does?  Does it allow multiple locks from the same thread?  It should, especially given that it is impossible to forget the unlock call (that in itself is a great boon to multithreaded development).
> 
> Yes it does.

Ahh, good to know.  I suspected it would but was a bit lazy by not testing it before I posted, thanks for the confirmation.

> > Another good idea is to provide a ThreadPool construct, do we have one of those floating around (pun intended).  The idea being that when you need a thread you ask the pool for one and it supplies it, then once you're done with it you release it back into the pool for the next piece of code to pick it up and run with it (get it, run with it, like a ball... well I thought it was funny).
> 
> Tango will almost certainly have one prior to its 1.0 release.  It already has a ThreadGroup object, but this is more for simply grouping multiple threads than it is for providing a general means for performing async. tasks.  I'm not yet sure just how extensive multi-threading support will be added to Tango by 1.0, but definitely more than it has now.

Good to know.  I'm just about to start a new job but once I settle in I might have some time to help out, if you want any.  I have sooo many hobbies (the latest of which is snooker! that I dont want to promise anything.)

Regan Heath