I would like to be able to spawn a thread, which does a lot of work, and then returns (in some way) the result of its computations to the main thread. I would really like to avoid copying its results if possible.

Logically, what I would like is something like this:


class LotsOfData { ... }

void doStuff(LotsOfData d) { ... }

shared(LotsOfData[]) data;

void main()
{
  spawn( &doWork );
  while(true)
  {
    foreach (d; data)
      doStuff(d);
  }
}

void doWork()
{
  LotsOfData d = new LotsOfData;
  // lots of computation on d
  data ~= d;
}


Essentially, the work that doWork does needs to be returned to the main thread asynchronously, and obviously in a thread-safe manner.

What's the best way to do this? The above won't work because ~= isn't atomic, so you can't do it on shared data.

Peter Alexander Wrote:

> Essentially, the work that doWork does needs to be returned to the main thread asynchronously, and obviously in a thread-safe manner.
> 
> What's the best way to do this? The above won't work because ~= isn't atomic, so you can't do it on shared data.

Disclaimer: I don't really do much with threading/concurrency.

You might look at using parallelfuture, the library to be added to Phobos as parallelism. An example similar to yours, though instead of the doWork getting a thread it is the doStuff function, and this example is for a call center:

https://gist.github.com/774983

Call => LoadsOfData
callGenerator => doWork
Threed.sleep => doStuff

You will probably also want to return an Immutable Call from callGenerator, that should result in a non-copy share (Though maybe it already does that).

You may also want to look at std.concurrency and make use of message passing.

https://gist.github.com/773979

Hopefully this are a good starting place.

On 02/04/2011 07:18 PM, Peter Alexander wrote:
> I would like to be able to spawn a thread, which does a lot of work, and then
> returns (in some way) the result of its computations to the main thread. I
> would really like to avoid copying its results if possible.
>
> Logically, what I would like is something like this:
>
>
> class LotsOfData { ... }
>
> void doStuff(LotsOfData d) { ... }
>
> shared(LotsOfData[]) data;
>
> void main()
> {
> spawn( &doWork );
> while(true)
> {
> foreach (d; data)
> doStuff(d);
> }
> }
>
> void doWork()
> {
> LotsOfData d = new LotsOfData;
> // lots of computation on d
> data ~= d;
> }
>
>
> Essentially, the work that doWork does needs to be returned to the main thread
> asynchronously, and obviously in a thread-safe manner.
>
> What's the best way to do this? The above won't work because ~= isn't atomic,
> so you can't do it on shared data.

(I have few exp in the paradigm, so don't believe me.)

It seems your problem is a typical case that cannot be safe as is. Essentially, IIUC, you want a shared set of data to be fed (more generally: mutated) from a thread, while another thread (here, the main one) processes bits of it. How can this be correct as is --except as you say if mutating operations were atomic?
I think in such a case you /must/ have a communication protocal between both tasks/threads. It is not due to language features (present ot not, this or that way) but to the problem itself. Correct me if I'm wrong, please.

Denis
-- 
_________________
vita es estrany
spir.wikidot.com

On 4/02/11 8:23 PM, Jesse Phillips wrote:
> Peter Alexander Wrote:
>
>> Essentially, the work that doWork does needs to be returned to the main
>> thread asynchronously, and obviously in a thread-safe manner.
>>
>> What's the best way to do this? The above won't work because ~= isn't
>> atomic, so you can't do it on shared data.
>
> Disclaimer: I don't really do much with threading/concurrency.
>
> You might look at using parallelfuture, the library to be added to Phobos as parallelism. An example similar to yours, though instead of the doWork getting a thread it is the doStuff function, and this example is for a call center:
>
> https://gist.github.com/774983
>
> Call =>  LoadsOfData
> callGenerator =>  doWork
> Threed.sleep =>  doStuff
>
> You will probably also want to return an Immutable Call from callGenerator, that should result in a non-copy share (Though maybe it already does that).
>
> You may also want to look at std.concurrency and make use of message passing.
>
> https://gist.github.com/773979
>
> Hopefully this are a good starting place.

Thanks, I'll take a look at parallelism.

How would you do it with message passing though? As I understand, all of the std.concurrency message passing routines are blocking, and I need this to be asynchronous.

spir Wrote:

> (I have few exp in the paradigm, so don't believe me.)
> 
> It seems your problem is a typical case that cannot be safe as is. Essentially, IIUC, you want a shared set of data to be fed (more generally: mutated) from a thread, while another thread (here, the main one) processes bits of it. How can this be correct as is --except as you say if mutating operations were atomic? I think in such a case you /must/ have a communication protocal between both tasks/threads. It is not due to language features (present ot not, this or that way) but to the problem itself. Correct me if I'm wrong, please.

Well I think that this is the model that goroutines lend them selves to. You have a producer input into a channel and a consumer on another thread/machine. To get such a behavior your don't need a language feature, but as of yet D does not have anything exactly comparable.

On 4/02/11 8:42 PM, spir wrote:
> On 02/04/2011 07:18 PM, Peter Alexander wrote:
>> I would like to be able to spawn a thread, which does a lot of work,
>> and then
>> returns (in some way) the result of its computations to the main
>> thread. I
>> would really like to avoid copying its results if possible.
>>
>> Logically, what I would like is something like this:
>>
>>
>> class LotsOfData { ... }
>>
>> void doStuff(LotsOfData d) { ... }
>>
>> shared(LotsOfData[]) data;
>>
>> void main()
>> {
>> spawn( &doWork );
>> while(true)
>> {
>> foreach (d; data)
>> doStuff(d);
>> }
>> }
>>
>> void doWork()
>> {
>> LotsOfData d = new LotsOfData;
>> // lots of computation on d
>> data ~= d;
>> }
>>
>>
>> Essentially, the work that doWork does needs to be returned to the
>> main thread
>> asynchronously, and obviously in a thread-safe manner.
>>
>> What's the best way to do this? The above won't work because ~= isn't
>> atomic,
>> so you can't do it on shared data.
>
> (I have few exp in the paradigm, so don't believe me.)
>
> It seems your problem is a typical case that cannot be safe as is.
> Essentially, IIUC, you want a shared set of data to be fed (more
> generally: mutated) from a thread, while another thread (here, the main
> one) processes bits of it. How can this be correct as is --except as you
> say if mutating operations were atomic?
> I think in such a case you /must/ have a communication protocal between
> both tasks/threads. It is not due to language features (present ot not,
> this or that way) but to the problem itself. Correct me if I'm wrong,
> please.
>
> Denis

You may be right, I don't know (that's why I'm asking!)

I was hoping that I would be able to do something using synchronized classes, but I'm not fully up to speed with how they interact with shared and immutable, and all the semantics associated with it. I understand that passing immutable value types around is very easy, but unfortunately that's rarely practical.

Things might be easier if the error messages associated with D's concurrent features weren't especially unhelpful (for example, trying to spawn a thread with reference type parameters just gives you a 'no match for spawn template' error). It's nice that it stops you from doing such things, but it would be nice if it told me why it's not going to let me do them.

Peter Alexander Wrote:
> 
> How would you do it with message passing though? As I understand, all of the std.concurrency message passing routines are blocking, and I need this to be asynchronous.

What do you mean by blocking?  The receive call will block until a message matching one of the supplied types arrives, but if you don't like this you can always use receiveTimeout.  send() doesn't deep copy objects, so the only reference types send() will currently accept are those labeled as shared or immutable (Unique!T will probably be added at some point, which is more appropriate for your situation).  So to use send() known unique reference data you'll have to cast to/from shared or immutable.  Nasty, but it'll work.

On 02/05/2011 12:21 AM, Peter Alexander wrote:
> Things might be easier if the error messages associated with D's concurrent
> features weren't especially unhelpful (for example, trying to spawn a thread
> with reference type parameters just gives you a 'no match for spawn template'
> error). It's nice that it stops you from doing such things, but it would be
> nice if it told me why it's not going to let me do them.

Yes, there was some times ago a thread about how to make template constraint errors more helpful.

Denis
-- 
_________________
vita es estrany
spir.wikidot.com

Peter Alexander Wrote:
> 
> Things might be easier if the error messages associated with D's concurrent features weren't especially unhelpful (for example, trying to spawn a thread with reference type parameters just gives you a 'no match for spawn template' error). It's nice that it stops you from doing such things, but it would be nice if it told me why it's not going to let me do them.

Could you provide an example?  When passing reference data, the error you should see is: "Aliases to mutable thread-local data not allowed."  It's a static assert inside send().

On 5/02/11 12:11 AM, Sean Kelly wrote:
> Peter Alexander Wrote:
>>
>> Things might be easier if the error messages associated with D's
>> concurrent features weren't especially unhelpful (for example, trying to
>> spawn a thread with reference type parameters just gives you a 'no match
>> for spawn template' error). It's nice that it stops you from doing such
>> things, but it would be nice if it told me why it's not going to let me
>> do them.
>
> Could you provide an example?  When passing reference data, the error you should see is: "Aliases to mutable thread-local data not allowed."  It's a static assert inside send().

Now that I've investigated a bit more, it appears to be unrelated to reference types, and instead was an error about using a nested function:

import std.concurrency;
void main()
{
  void foo() {}
  spawn(&foo);
}

---
test.d(5): Error: template std.concurrency.spawn(T...) does not match any function template declaration
test.d(5): Error: template std.concurrency.spawn(T...) cannot deduce template function from argument types !()(void delegate())
---

Why does it think that the function is a delegate?