Hi guys,
I know this will have probably been answered before, but does D have a concept similar to Go's channels when it comes to tasks?

I believe (according to a few forum posts hinting at it) that such a thing exists for threads, but not for the lightweight tasks provided by std.parallelism. Are there any plans to add such a feature to the library?

If not, how would I go about implementing them in D? I imagine I would use some form of blocking list, but  I've not done much research on how they are done in Go/Rust/etc.

Thanks,
Matt.

P.S. Sorry if this is in the wrong sub-forum, I'll be happy for mods to move it if necessary.

On Thursday, 18 July 2013 at 15:29:00 UTC, Matt wrote:
> Hi guys,
> I know this will have probably been answered before, but does D have a concept similar to Go's channels when it comes to tasks?

I don't think this is really an answer, but awhile back I was looking into the difference of D and Go for parallel/concurrency and created these two examples[1][2] from the blog

http://www.mprescient.com/journal/2011/1/9/concurrency-in-go-a-call-center-tutorial.html

1. https://gist.github.com/JesseKPhillips/773979
2. https://gist.github.com/JesseKPhillips/774983

The first is a translation to message passing, the second is using std.parallelism

I think there is room for higher level APIs, but I don't think we will see anything that resembles Goroutines being added.

> P.S. Sorry if this is in the wrong sub-forum, I'll be happy for mods to move it if necessary.

Should go to D.learn, but not a huge deal.

On Jul 18, 2013, at 8:28 AM, Matt <MATTCA@sky.com> wrote:

> Hi guys,
> I know this will have probably been answered before, but does D have a concept similar to Go's channels when it comes to tasks?
> 
> I believe (according to a few forum posts hinting at it) that such a thing exists for threads, but not for the lightweight tasks provided by std.parallelism. Are there any plans to add such a feature to the library?

Not as such.  I'd like to make Fibers each have their own message queue in std.concurrency, but that means making TLS work at the fiber level, which is tricky.

I think there is value in the CSP model (ie. channels), but haven't done anything about it in terms of library work.

On Thursday, 18 July 2013 at 16:41:34 UTC, Jesse Phillips wrote:
> I don't think this is really an answer, but awhile back I was looking into the difference of D and Go for parallel/concurrency and created these two examples[1][2] from the blog
>
> http://www.mprescient.com/journal/2011/1/9/concurrency-in-go-a-call-center-tutorial.html
>
> 1. https://gist.github.com/JesseKPhillips/773979
> 2. https://gist.github.com/JesseKPhillips/774983
>
> The first is a translation to message passing, the second is using std.parallelism
>
> I think there is room for higher level APIs, but I don't think we will see anything that resembles Goroutines being added.

Thanks for the examples, I'll have to take a look at them later! I guess the different backgrounds is the reason for a lack of D 'goroutines', but parallelism seemed a good alternative.

Is the performance of parallelism and it's taskpools similar to goroutines? It seems they both do the same thing (adding the tasks to a queue, allocating them to a group of threads, etc), but does the level of implementation (library vs language) mean a speed difference?

Furthermore, is there a limit to the number of tasks that can be spawned and performed in parallel? I know Rust allows for hundreds, even thousands, of tasks to be executed "concurrently."

Thanks for the replies,
Matt.

On Thursday, 18 July 2013 at 17:55:36 UTC, Sean Kelly wrote:
> Not as such.  I'd like to make Fibers each have their own message queue in std.concurrency, but that means making TLS work at the fiber level, which is tricky.
>
> I think there is value in the CSP model (ie. channels), but haven't done anything about it in terms of library work.

From what I've read about Fibers, it seems they are more similar to what I'm looking for (a post on stackoverflow implied they are similar to Erlang's implementation of lightweight tasks). Does this mean to say they are similar to the tasks provided by std.parallelism?

Thanks for the reply,
Matt.

On Thursday, 18 July 2013 at 18:38:57 UTC, Matt wrote:
> On Thursday, 18 July 2013 at 17:55:36 UTC, Sean Kelly wrote:
>> Not as such.  I'd like to make Fibers each have their own message queue in std.concurrency, but that means making TLS work at the fiber level, which is tricky.
>>
>> I think there is value in the CSP model (ie. channels), but haven't done anything about it in terms of library work.
>
> From what I've read about Fibers, it seems they are more similar to what I'm looking for (a post on stackoverflow implied they are similar to Erlang's implementation of lightweight tasks). Does this mean to say they are similar to the tasks provided by std.parallelism?
>
> Thanks for the reply,
> Matt.

vibe.d is based on fibers, you might want to check it out.
http://vibed.org/features#fibers

On Jul 18, 2013, at 11:38 AM, "Matt" <MATTCA@sky.com> wrote:

> On Thursday, 18 July 2013 at 17:55:36 UTC, Sean Kelly wrote:
>> Not as such.  I'd like to make Fibers each have their own message queue in std.concurrency, but that means making TLS work at the fiber level, which is tricky.
>> 
>> I think there is value in the CSP model (ie. channels), but haven't done anything about it in terms of library work.
> 
> From what I've read about Fibers, it seems they are more similar to what I'm looking for (a post on stackoverflow implied they are similar to Erlang's implementation of lightweight tasks). Does this mean to say they are similar to the tasks provided by std.parallelism?

Functionally, fibers are coroutines.  They have their own stack and execute within the context of the calling thread.  Most languages that scale to thousands or millions of concurrent threads/processes use fibers (often called "green threads" in this context) to pull it off.  The obstacle for D is that global data is thread-local by default, so if we were to use fibers in the back-end to allow the creation of a large number of "threads", fibers would basically need their own TLS to avoid breaking code.  We already do in-library TLS for OSX and so it shouldn't be terribly difficult to use this logic for fibers, but things get tricky once you consider dynamic libraries.  It really should happen at some point though, for std.concurrency to operate at its full potential.

On 2013-07-18 21:08, Sean Kelly wrote:

> We already do in-library TLS for OSX and so it shouldn't be terribly difficult to use this logic for fibers, but things get tricky once you consider dynamic libraries.

To be able to support dynamic libraries and TLS on Mac OS X we will most likely need to switch to the native implementation of TLS, supported in Mac OS X 10.7 and later. Or we would have to copy/mimic the TLS related code from the dynamic loader into druntime.

-- 
/Jacob Carlborg

On Thursday, 18 July 2013 at 17:55:36 UTC, Sean Kelly wrote:
> On Jul 18, 2013, at 8:28 AM, Matt <MATTCA@sky.com> wrote:
>
>> Hi guys,
>> I know this will have probably been answered before, but does D have a concept similar to Go's channels when it comes to tasks?
>> 
>> I believe (according to a few forum posts hinting at it) that such a thing exists for threads, but not for the lightweight tasks provided by std.parallelism. Are there any plans to add such a feature to the library?
>
> Not as such.  I'd like to make Fibers each have their own message queue in std.concurrency, but that means making TLS work at the fiber level, which is tricky.
>

Yes please yes.

Did I said yes yet ?

> I think there is value in the CSP model (ie. channels), but haven't done anything about it in terms of library work.

On Thursday, 18 July 2013 at 19:08:25 UTC, Sean Kelly wrote:
> Functionally, fibers are coroutines.  They have their own stack and execute within the context of the calling thread.  Most languages that scale to thousands or millions of concurrent threads/processes use fibers (often called "green threads" in this context) to pull it off.  The obstacle for D is that global data is thread-local by default, so if we were to use fibers in the back-end to allow the creation of a large number of "threads", fibers would basically need their own TLS to avoid breaking code.  We already do in-library TLS for OSX and so it shouldn't be terribly difficult to use this logic for fibers, but things get tricky once you consider dynamic libraries.  It really should happen at some point though, for std.concurrency to operate at its full potential.

My take is that we should keep fibers as they are. But change Thread (as the class in the runtime) to be Fibers with FLS and then let the runtime operate system thread and schedule Thread on them.

The read system thread can still be provided in some dark corner of the standard lib or runtime.