I want to applaud Sean Kelly and everyone who worked on std.concurrency for a great API, and wish that I could easily write Cocoa applications with it.

I'm writing a screen recording program in Objective-C, and to make sure each frame has an equal length, I have two threads: one that takes the screenshot at certain intervals, the other that assembles it into a quicktime movie. After writing an implementation of a thread-safe queue, where the picture-taking thread adds and the assembling thread removes, I realized the huge similarity between what I'd done and std.concurrency.

The std.concurrency module would have made hours of effort in ObjC take five minutes in D. Well done!

Thanks :-)  If you have ideas on how it could be improved, please let me know.

On Jan 19, 2012, at 12:58 PM, Nathan M. Swan wrote:

> I want to applaud Sean Kelly and everyone who worked on std.concurrency for a great API, and wish that I could easily write Cocoa applications with it.
> 
> I'm writing a screen recording program in Objective-C, and to make sure each frame has an equal length, I have two threads: one that takes the screenshot at certain intervals, the other that assembles it into a quicktime movie. After writing an implementation of a thread-safe queue, where the picture-taking thread adds and the assembling thread removes, I realized the huge similarity between what I'd done and std.concurrency.
> 
> The std.concurrency module would have made hours of effort in ObjC take five minutes in D. Well done!

On 2012-01-19 21:58, Nathan M. Swan wrote:
> I want to applaud Sean Kelly and everyone who worked on std.concurrency
> for a great API, and wish that I could easily write Cocoa applications
> with it.
>
> I'm writing a screen recording program in Objective-C, and to make sure
> each frame has an equal length, I have two threads: one that takes the
> screenshot at certain intervals, the other that assembles it into a
> quicktime movie. After writing an implementation of a thread-safe queue,
> where the picture-taking thread adds and the assembling thread removes,
> I realized the huge similarity between what I'd done and std.concurrency.
>
> The std.concurrency module would have made hours of effort in ObjC take
> five minutes in D. Well done!

I don't know if it's the same but doesn't Objective-C makes that quite simple as well since Mac OS X 10.6 using grand central dispatch:

http://en.wikipedia.org/wiki/Grand_Central_Dispatch

-- 
/Jacob Carlborg

On 20 January 2012 00:36, Sean Kelly <sean@invisibleduck.org> wrote:

> Thanks :-)  If you have ideas on how it could be improved, please let me know.
>
> On Jan 19, 2012, at 12:58 PM, Nathan M. Swan wrote:
>
> > I want to applaud Sean Kelly and everyone who worked on std.concurrency
> for a great API, and wish that I could easily write Cocoa applications with it.
> >
> > I'm writing a screen recording program in Objective-C, and to make sure
> each frame has an equal length, I have two threads: one that takes the screenshot at certain intervals, the other that assembles it into a quicktime movie. After writing an implementation of a thread-safe queue, where the picture-taking thread adds and the assembling thread removes, I realized the huge similarity between what I'd done and std.concurrency.
> >
> > The std.concurrency module would have made hours of effort in ObjC take
> five minutes in D. Well done!
>

I had some troubles with std.concurrency which I thought it might be nice to address.

Perhaps the most major problem I had was related to the concept of thread
ownership. If a spawned threads parent thread dies, it also receives a
signal to kill its self, but it seems impossible to reassign ownership.
In my case I had threads A, B and C...
  Thread A is the main thread, which may spawn many temporary worker
threads B, which may last 1-2 seconds.
  During the life of B, it may spawn C, which may persist for hours.
  B promptly dies, and any C's that were spawned receive the kill signal,
which I ignore.
  Thread A, the main thread may exit, and I would really like all C's to
receive that notification, but they were are all orphaned when their B died.
The problem is, when I spawn a C, it should be created as a child of A
somehow, rather than a child of the transient B... Some mechanism to solve
this sort of problem would be useful.

Another usability problem I had was that, intuitively, the simpler function
with intuitive usage pattern receiveOnly() should be named receive().
And receive() with the complex var-arg list of delegates should be named
something more complex.
receiveTimeout() has no receiveTimeoutOnly(), which is the function I
almost always want to use... and receiveTimeout() didn't actually work for
me anyway (it wouldn't receive a duration as per the documentation, I had
to pass an int)

I wonder if there is a solution to the 'shared' problem. Basically every
single line of code that uses the std.concurrenty api is riddled with casts
to/from shared... really ugly.
I think the problem here is that I'm not SHARING values between threads,
I'm actually passing ownership of something to another thread. So I wonder
if some improvement can be made in this area.

I was going to complain about the documentation, but I just checked, and it seems to have had work since I was reading it. Looks much better now! :)

On Friday, 20 January 2012 at 13:10:37 UTC, Manu wrote:
> On 20 January 2012 00:36, Sean Kelly <sean@invisibleduck.org> wrote:
>
>> Thanks :-)  If you have ideas on how it could be improved, please let me
>> know.
>>
>> On Jan 19, 2012, at 12:58 PM, Nathan M. Swan wrote:
>>
>> > I want to applaud Sean Kelly and everyone who worked on std.concurrency
>> for a great API, and wish that I could easily write Cocoa applications with
>> it.
>> >
>> > I'm writing a screen recording program in Objective-C, and to make sure
>> each frame has an equal length, I have two threads: one that takes the
>> screenshot at certain intervals, the other that assembles it into a
>> quicktime movie. After writing an implementation of a thread-safe queue,
>> where the picture-taking thread adds and the assembling thread removes, I
>> realized the huge similarity between what I'd done and std.concurrency.
>> >
>> > The std.concurrency module would have made hours of effort in ObjC take
>> five minutes in D. Well done!
>>
>
> I had some troubles with std.concurrency which I thought it might be nice
> to address.
>
> Perhaps the most major problem I had was related to the concept of thread
> ownership. If a spawned threads parent thread dies, it also receives a
> signal to kill its self, but it seems impossible to reassign ownership.
> In my case I had threads A, B and C...
> Thread A is the main thread, which may spawn many temporary worker
> threads B, which may last 1-2 seconds.
> During the life of B, it may spawn C, which may persist for hours.
> B promptly dies, and any C's that were spawned receive the kill signal,
> which I ignore.
> Thread A, the main thread may exit, and I would really like all C's to
> receive that notification, but they were are all orphaned when their B died.
> The problem is, when I spawn a C, it should be created as a child of A
> somehow, rather than a child of the transient B... Some mechanism to solve
> this sort of problem would be useful.
>
> Another usability problem I had was that, intuitively, the simpler function
> with intuitive usage pattern receiveOnly() should be named receive().
> And receive() with the complex var-arg list of delegates should be named
> something more complex.
> receiveTimeout() has no receiveTimeoutOnly(), which is the function I
> almost always want to use... and receiveTimeout() didn't actually work for
> me anyway (it wouldn't receive a duration as per the documentation, I had
> to pass an int)
>
> I wonder if there is a solution to the 'shared' problem. Basically every
> single line of code that uses the std.concurrenty api is riddled with casts
> to/from shared... really ugly.
> I think the problem here is that I'm not SHARING values between threads,
> I'm actually passing ownership of something to another thread. So I wonder
> if some improvement can be made in this area.
>
> I was going to complain about the documentation, but I just checked, and it
> seems to have had work since I was reading it. Looks much better now! :)

It still needs to document that the ops in "void receive(T...)(T ops)" can return false if they did not handle the message. This is useful when you want to receive a message from a specific spawned subthread (tid).

/Jonas

On Jan 20, 2012, at 5:10 AM, Manu wrote:
> 
> I had some troubles with std.concurrency which I thought it might be nice to address.
> 
> Perhaps the most major problem I had was related to the concept of thread ownership. If a spawned threads parent thread dies, it also receives a signal to kill its self, but it seems impossible to reassign ownership.
> In my case I had threads A, B and C...
>   Thread A is the main thread, which may spawn many temporary worker threads B, which may last 1-2 seconds.
>   During the life of B, it may spawn C, which may persist for hours.
>   B promptly dies, and any C's that were spawned receive the kill signal, which I ignore.
>   Thread A, the main thread may exit, and I would really like all C's to receive that notification, but they were are all orphaned when their B died.
> The problem is, when I spawn a C, it should be created as a child of A somehow, rather than a child of the transient B... Some mechanism to solve this sort of problem would be useful.

Erlang has functions to link and unlink threads from one another, and I've already implemented a bit of it in std.concurrency.  A fuller implementation would probably be sufficient for your needs.


> Another usability problem I had was that, intuitively, the simpler function with intuitive usage pattern receiveOnly() should be named receive().
> And receive() with the complex var-arg list of delegates should be named something more complex.

Matter of opinion.  I think receiveOnly indicates what the function does, and I think it's important to indicate that receiving anything else is unexpected and will trigger an error.


> receiveTimeout() has no receiveTimeoutOnly(), which is the function I almost always want to use... and receiveTimeout() didn't actually work for me anyway (it wouldn't receive a duration as per the documentation, I had to pass an int)

I think this is an artifact of the syntax.  There's no way to have a return value that indicates a timeout occurred.  If receiving a duration (ie. a non-reference type) doesn't work then I'd consider that a bug.


> I wonder if there is a solution to the 'shared' problem. Basically every single line of code that uses the std.concurrenty api is riddled with casts to/from shared... really ugly.

I think Unique!T should work for the sending side, and then the receive side wouldn't have to receive shared(T).  It's something that's been on the radar, but I haven't done anything about it yet.

On 20 January 2012 20:06, Sean Kelly <sean@invisibleduck.org> wrote:

> On Jan 20, 2012, at 5:10 AM, Manu wrote:
> >
> > I had some troubles with std.concurrency which I thought it might be
> nice to address.
> >
> > Perhaps the most major problem I had was related to the concept of
> thread ownership. If a spawned threads parent thread dies, it also receives a signal to kill its self, but it seems impossible to reassign ownership.
> > In my case I had threads A, B and C...
> >   Thread A is the main thread, which may spawn many temporary worker
> threads B, which may last 1-2 seconds.
> >   During the life of B, it may spawn C, which may persist for hours.
> >   B promptly dies, and any C's that were spawned receive the kill
> signal, which I ignore.
> >   Thread A, the main thread may exit, and I would really like all C's to
> receive that notification, but they were are all orphaned when their B died.
> > The problem is, when I spawn a C, it should be created as a child of A
> somehow, rather than a child of the transient B... Some mechanism to solve this sort of problem would be useful.
>
> Erlang has functions to link and unlink threads from one another, and I've already implemented a bit of it in std.concurrency.  A fuller implementation would probably be sufficient for your needs.
>
>
> > Another usability problem I had was that, intuitively, the simpler
> function with intuitive usage pattern receiveOnly() should be named
> receive().
> > And receive() with the complex var-arg list of delegates should be named
> something more complex.
>
> Matter of opinion.  I think receiveOnly indicates what the function does, and I think it's important to indicate that receiving anything else is unexpected and will trigger an error.
>

Well I'm just saying as a new comer what I consider to be intuitive or not
with respect to what I've used in other languages (in my case, C/C++, C#,
java, php, ecma, etc)... I've never seen an api like receive() in any
language before. It definitely is a D-ish API, and wouldn't be expected by
most users.
I also think most instances of passing messages between 2 threads will have
a tight well-defined expectation of send/receive types. receiveOnly() seems
far more intuitive to me, and simpler to achieve 90(100?)% of my jobs.
Eg, I press '.' and the list of methods appears, and I skim through the
list and choose the one that looks appropriate, I'll choose receive, and
then I'll be puzzled by the argument list and why it doesn't work like I
expect, after a little wasted time, I may begrudgingly read the manual... I
personally feel this is an API failure, and the single most important thing
that C# gets right. You can literally code C# effectively with absolutely
no prior knowledge of the language just using the '.' key with
code-complete in your IDE. The API's are really exceptionally intuitive.



> > receiveTimeout() has no receiveTimeoutOnly(), which is the function I
> almost always want to use... and receiveTimeout() didn't actually work for
> me anyway (it wouldn't receive a duration as per the documentation, I had
> to pass an int)
>
> I think this is an artifact of the syntax.  There's no way to have a return value that indicates a timeout occurred.  If receiving a duration (ie. a non-reference type) doesn't work then I'd consider that a bug.
>

Syntax or otherwise, it's kinda broken. receiveOnlyTimeout() is a very important function for my money. I think many usages will not want to block indefinitely.

Actually, on that note, I really wanted a poll() function. I never want to
block for messages, just grab one if there is one waiting in the queue.
receiveTimeout(0) seems to be the only way to do it... Again, not
particularly nice or intuitive.
Consider the Win32 message loop: PeekMessage/GetMessage/WaitMessage.. I
think they're all good to have.


> I wonder if there is a solution to the 'shared' problem. Basically every single line of code that uses the std.concurrenty api is riddled with casts to/from shared... really ugly.
>
> I think Unique!T should work for the sending side, and then the receive side wouldn't have to receive shared(T).  It's something that's been on the radar, but I haven't done anything about it yet.


Cool, I'll be interested to see what nice solutions for this could exist.

Manu wrote:
> Eg, I press '.' and the list of methods appears, and I skim through the
> list and choose the one that looks appropriate, I'll choose receive, and
> then I'll be puzzled by the argument list and why it doesn't work like I
> expect, after a little wasted time, I may begrudgingly read the manual... I
> personally feel this is an API failure, and the single most important thing
> that C# gets right. You can literally code C# effectively with absolutely
> no prior knowledge of the language just using the '.' key with
> code-complete in your IDE. The API's are really exceptionally intuitive.

This is a big restraint to D's popularity. It's certainly a complaint I've heard from others. An IDE with intelligence might have been a luxury in the past, but it's quickly becoming essential to large project development. Things like hunting through poorly cross-referenced documentation just to find out how to convert a string to an int, then doing it all over again when you realize the same function doesn't go both ways is just a pain in the ass.

On 21 January 2012 17:23, F i L <witte2008@gmail.com> wrote:

> Manu wrote:
>
>> Eg, I press '.' and the list of methods appears, and I skim through the
>> list and choose the one that looks appropriate, I'll choose receive, and
>> then I'll be puzzled by the argument list and why it doesn't work like I
>> expect, after a little wasted time, I may begrudgingly read the manual...
>> I
>> personally feel this is an API failure, and the single most important
>> thing
>> that C# gets right. You can literally code C# effectively with absolutely
>> no prior knowledge of the language just using the '.' key with
>> code-complete in your IDE. The API's are really exceptionally intuitive.
>>
>
> This is a big restraint to D's popularity. It's certainly a complaint I've heard from others. An IDE with intelligence might have been a luxury in the past, but it's quickly becoming essential to large project development. Things like hunting through poorly cross-referenced documentation just to find out how to convert a string to an int, then doing it all over again when you realize the same function doesn't go both ways is just a pain in the ass.
>

'quickly becoming' :) .. I think that happened 5 years ago. It's long been a basic requirement, and does really need some attention.

I suggest checking out Erlang messaging, as it's the basis for this design. Maybe then things will be a bit clearer.

Sent from my iPhone

On Jan 21, 2012, at 6:34 AM, Manu <turkeyman@gmail.com> wrote:

> On 20 January 2012 20:06, Sean Kelly <sean@invisibleduck.org> wrote: On Jan 20, 2012, at 5:10 AM, Manu wrote:
> >
> > I had some troubles with std.concurrency which I thought it might be nice to address.
> >
> > Perhaps the most major problem I had was related to the concept of thread ownership. If a spawned threads parent thread dies, it also receives a signal to kill its self, but it seems impossible to reassign ownership.
> > In my case I had threads A, B and C...
> >   Thread A is the main thread, which may spawn many temporary worker threads B, which may last 1-2 seconds.
> >   During the life of B, it may spawn C, which may persist for hours.
> >   B promptly dies, and any C's that were spawned receive the kill signal, which I ignore.
> >   Thread A, the main thread may exit, and I would really like all C's to receive that notification, but they were are all orphaned when their B died.
> > The problem is, when I spawn a C, it should be created as a child of A somehow, rather than a child of the transient B... Some mechanism to solve this sort of problem would be useful.
> 
> Erlang has functions to link and unlink threads from one another, and I've already implemented a bit of it in std.concurrency.  A fuller implementation would probably be sufficient for your needs.
> 
> 
> > Another usability problem I had was that, intuitively, the simpler function with intuitive usage pattern receiveOnly() should be named receive().
> > And receive() with the complex var-arg list of delegates should be named something more complex.
> 
> Matter of opinion.  I think receiveOnly indicates what the function does, and I think it's important to indicate that receiving anything else is unexpected and will trigger an error.
> 
> Well I'm just saying as a new comer what I consider to be intuitive or not with respect to what I've used in other languages (in my case, C/C++, C#, java, php, ecma, etc)... I've never seen an api like receive() in any language before. It definitely is a D-ish API, and wouldn't be expected by most users.
> I also think most instances of passing messages between 2 threads will have a tight well-defined expectation of send/receive types. receiveOnly() seems far more intuitive to me, and simpler to achieve 90(100?)% of my jobs.
> Eg, I press '.' and the list of methods appears, and I skim through the list and choose the one that looks appropriate, I'll choose receive, and then I'll be puzzled by the argument list and why it doesn't work like I expect, after a little wasted time, I may begrudgingly read the manual... I personally feel this is an API failure, and the single most important thing that C# gets right. You can literally code C# effectively with absolutely no prior knowledge of the language just using the '.' key with code-complete in your IDE. The API's are really exceptionally intuitive.
> 
> 
> > receiveTimeout() has no receiveTimeoutOnly(), which is the function I almost always want to use... and receiveTimeout() didn't actually work for me anyway (it wouldn't receive a duration as per the documentation, I had to pass an int)
> 
> I think this is an artifact of the syntax.  There's no way to have a return value that indicates a timeout occurred.  If receiving a duration (ie. a non-reference type) doesn't work then I'd consider that a bug.
> 
> Syntax or otherwise, it's kinda broken. receiveOnlyTimeout() is a very important function for my money. I think many usages will not want to block indefinitely.
> 
> Actually, on that note, I really wanted a poll() function. I never want to block for messages, just grab one if there is one waiting in the queue. receiveTimeout(0) seems to be the only way to do it... Again, not particularly nice or intuitive.
> Consider the Win32 message loop: PeekMessage/GetMessage/WaitMessage.. I think they're all good to have.
> 
> 
> > I wonder if there is a solution to the 'shared' problem. Basically every single line of code that uses the std.concurrenty api is riddled with casts to/from shared... really ugly.
> 
> I think Unique!T should work for the sending side, and then the receive side wouldn't have to receive shared(T).  It's something that's been on the radar, but I haven't done anything about it yet.
> 
> Cool, I'll be interested to see what nice solutions for this could exist.