On Tuesday, 6 August 2024 at 02:20:52 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 06/08/2024 3:50 AM, Sebastiaan Koppe wrote:
>> On Monday, 5 August 2024 at 01:36:31 UTC, Richard (Rikki) Andrew Cattermole wrote:
>>> Coroutines is a stack machine transformed representation for a function. It enables storing the state of a function externally to the stack to allow for high throughput event handling.
>> 
>> Thank you for spearheading this. Having coroutines in the language would be a big step forward, and hopefully pave the way for more non-blocking programming in D.
>> 
>> There are a few points I would like to bring up early though.
>> 
>> - I see no mention of C++'s coroutines. I think it would be good to learn from their design and implementation.
>>
>> - As you might know I am a big proponent of C++'s Senders/Receivers (a.k.a. P2300). One awesome integration they have is that Senders can be awaited by coroutines, and coroutines can await Senders. This allows for users to pick their preference, e.g. use convenient coroutines but suffer some allocation costs, and use Senders for more performant sections if need be. See
>> https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2300r7.html#design-awaitables-are-senders and https://ericniebler.com/2024/02/04/what-are-senders-good-for-anyway/
>
> My conclusion about P2300 is that it is all library code. This DIP would allow you to implement it.

https://github.com/symmetryinvestments/concurrency already implements a fairly large chunk of it.

I suppose what you mean is that this DIP would allow me to integrate coroutines with it.

>> - I don't understand the choice for `@async`. Sometimes resembling the word "coroutine" would be better in my opinion.
>
> It was ``@co`` for a long time.
>
> I had a number of people request ``async`` instead.
>
>> - In `while(Future!string readLine = socket.readUntil("\n"))` I suspect it to do an explicit await. How does that work?
>
> When a coroutine is acquired, it'll yield automatically, it is implicit.

What do you mean with 'acquired'?

>> - I find the use of `Future` a bit confusing. Futures generally carry too much synchronisation overhead with them and I doubt coroutines need a full Future implementation anyway. Perhaps call it a Task like they do in C++?
>
> This DIP does not propose library code. You are free to write whatever library code you want and have it construct itself from the language representation of a coroutine.

I fail to see what it then _does_ provide.

> This is a key design goal as there is a good chance we'll want different solutions for different tasks.

It ultimately needs to integrate with the language coroutine support, so I don't understand how it can be completely separate. And if it can't, then the language needs to provide some low-level interface to it, which, granted, libraries can extend on.

>> - The use of `@isasync` is a bit too cute for me. It also hides the fact sometime is a coroutine.
>
> I want to be placing a lot more emphasis on UDA's like ``@Route``, what we do right now with reflection is far too costly in terms of how to register symbols.

I think it distracts from the proposal. Likely `@Route` will be implemented in a library anyway.

> But yes, I do want to hide that it is a coroutine. The average person shouldn't have to care if its asynchronous of synchronous, it does not enable them to archive business goals faster.

I very much like the low-level/high-level proposed approach to Phobos3. I very much do not want the language to hide coroutine details, and instead want to be in full control, when and if a coroutine is called/yielded to.

>> - Instead of allowing a coroutine in a function that is not a coroutine itself - and injecting a blocking call - I would require explicit call to evaluate the coroutine instead. No magic.
>
> And that is library code ;)
>
> But yes, I did try to explain that you did not need language integration for this. In fact the prime sieve example show cases exactly what you suggest!

I do not understand what it does - and how! It is unclear to me how `&generate` is turned into a `InstantiableCoroutine!(int)`, how `makeInstance` works (or what it even does) and what `ch.block` does.

I can guess of course, but that explanation needs to be part of the DIP.

On 24/09/2024 10:47 PM, Sebastiaan Koppe wrote:
> On Tuesday, 6 August 2024 at 02:20:52 UTC, Richard (Rikki) Andrew Cattermole wrote:
>> On 06/08/2024 3:50 AM, Sebastiaan Koppe wrote:
>>> On Monday, 5 August 2024 at 01:36:31 UTC, Richard (Rikki) Andrew Cattermole wrote:
>>>> Coroutines is a stack machine transformed representation for a function. It enables storing the state of a function externally to the stack to allow for high throughput event handling.
>>>
>>> Thank you for spearheading this. Having coroutines in the language would be a big step forward, and hopefully pave the way for more non-blocking programming in D.
>>>
>>> There are a few points I would like to bring up early though.
>>>
>>> - I see no mention of C++'s coroutines. I think it would be good to learn from their design and implementation.
>>>
>>> - As you might know I am a big proponent of C++'s Senders/Receivers (a.k.a. P2300). One awesome integration they have is that Senders can be awaited by coroutines, and coroutines can await Senders. This allows for users to pick their preference, e.g. use convenient coroutines but suffer some allocation costs, and use Senders for more performant sections if need be. See
>>> https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2300r7.html#design-awaitables-are-senders and https://ericniebler.com/2024/02/04/what-are-senders-good-for-anyway/
>>
>> My conclusion about P2300 is that it is all library code. This DIP would allow you to implement it.
> 
> https://github.com/symmetryinvestments/concurrency already implements a fairly large chunk of it.
> 
> I suppose what you mean is that this DIP would allow me to integrate coroutines with it.

Yeah, the specific library isn't so important. It is up to you how you translate the coroutine descriptor into your library. The hooks are in place so there is no hard coded library code required.

>>> - I don't understand the choice for `@async`. Sometimes resembling the word "coroutine" would be better in my opinion.
>>
>> It was ``@co`` for a long time.
>>
>> I had a number of people request ``async`` instead.
>>
>>> - In `while(Future!string readLine = socket.readUntil("\n"))` I suspect it to do an explicit await. How does that work?
>>
>> When a coroutine is acquired, it'll yield automatically, it is implicit.
> 
> What do you mean with 'acquired'?

Stored in a variable.

On that note, I'm wondering if this is a little too limited, even to begin with.

A built-in to language "method" called ``yield`` on a coroutine library type, might be a good way to force a yield.

Yields should probably only happen if methods annotated as requiring a yield is accessed.

Needs thinking about.

>>> - I find the use of `Future` a bit confusing. Futures generally carry too much synchronisation overhead with them and I doubt coroutines need a full Future implementation anyway. Perhaps call it a Task like they do in C++?
>>
>> This DIP does not propose library code. You are free to write whatever library code you want and have it construct itself from the language representation of a coroutine.
> 
> I fail to see what it then _does_ provide.

The language feature, that then gets sliced and diced by the compiler into a descriptor, that then gets taken by library code into their own representation.

Library code cannot slice and dice a function up into a coroutine it has to be compiler backed.

See "Example Descriptor Object" https://gist.github.com/rikkimax/fe2578e1dfbf66346201fd191db4bdd4/0bb4442c092e061d520c35a43278f0819666f26f#example-descriptor-object

>> This is a key design goal as there is a good chance we'll want different solutions for different tasks.
> 
> It ultimately needs to integrate with the language coroutine support, so I don't understand how it can be completely separate. And if it can't, then the language needs to provide some low-level interface to it, which, granted, libraries can extend on.

Recognizing a library type as a coroutine, yielding, multiple returns ext. are all integrated yes.

What I was talking about here is related to data processing vs event processing vs whatever task people come up with. They have different needs to give the best user experience.

My focus here is to get enough to make it work for event processing as that is the one I have experience in with my library code.

>>> - The use of `@isasync` is a bit too cute for me. It also hides the fact sometime is a coroutine.
>>
>> I want to be placing a lot more emphasis on UDA's like ``@Route``, what we do right now with reflection is far too costly in terms of how to register symbols.
> 
> I think it distracts from the proposal. Likely `@Route` will be implemented in a library anyway.

I did not define a ``Route`` UDA. Only ``isasync`` to make library code like that work.

>> But yes, I do want to hide that it is a coroutine. The average person shouldn't have to care if its asynchronous of synchronous, it does not enable them to archive business goals faster.
> 
> I very much like the low-level/high-level proposed approach to Phobos3. I very much do not want the language to hide coroutine details, and instead want to be in full control, when and if a coroutine is called/yielded to.

I understand that you want this.

A design goal is the average programmer does not need to care about the function they are writing is a coroutine.

They should be able to slap ``@Route`` on their function, ask for some data and get it back without them having to do anything special.

This is the target audience I am designing this for.

If we need to allow the advanced user the ability to disable automatic yielding, then we can do that with a UDA later on. It does not need to be in a DIP nor be done now.

>>> - Instead of allowing a coroutine in a function that is not a coroutine itself - and injecting a blocking call - I would require explicit call to evaluate the coroutine instead. No magic.
>>
>> And that is library code ;)
>>
>> But yes, I did try to explain that you did not need language integration for this. In fact the prime sieve example show cases exactly what you suggest!
> 
> I do not understand what it does - and how! It is unclear to me how `&generate` is turned into a `InstantiableCoroutine!(int)`, how `makeInstance` works (or what it even does) and what `ch.block` does.
> 
> I can guess of course, but that explanation needs to be part of the DIP.

They are not provided by the DIP, these are examples that are based off my own codebase.

Note: they are in fact defined (except ``makeInstance`` as the name is self explanatory).

"The following template inference must work:"

```d
struct InstantiableCoroutine(Return, Args...) {
	static InstiableCoroutine opConstructCo(CoroutineDescriptor : __descriptorco)(CoroutineDescriptor) {
		return InstatiableCoroutine.init;
	}
}

void acceptCo(Return, Arguments...)(InstatiableCoroutine!(Return, Arguments) co) {
	static assert(Arguments.length == 1);
	static assert(is(Arguments[0] == int));
	static assert(is(Return == string));
}

acceptCo((int value) {
	return "hello!";
});
```

"This is described due to the introduction of the implicitly constructing operator overload function opConstructCo which may not be friendly towards template inference."

How you implement ``opConstructCo`` is specific to your library. It has no place in the DIP.

"It can also be used to construct a coroutine object using variable assignment or by returning a function with a typed coroutine library type on the outer function."

```d
void main() {
	InstantiableCoroutine!(int, int, int) var = &myCo;
}

int myCo(int a, int b) @async {
	return a + b;
}

InstantiableCoroutine!(int, int, int) returnedCo() {
	return (int a, int b) {
		return a + b;
	};
}
```

I can see that ``block`` while what it does is described all throughout, it is not spelled out in the example. Okay examples need more text describing them!

From what I gather from you reply in general, you are trying to find a library proposal where there is only a language one. It'll need some rework to hopefully stop that.

On Tuesday, 6 August 2024 at 02:00:39 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 06/08/2024 6:11 AM, Sebastiaan Koppe wrote:
>> It is similar to the approach Go takes in that it makes calling C functions more difficult though.
>
> Go goes a bit further than that, their goroutines are effectively in their own calling convention.
>
> We cannot replicate what they did, as we have strict requirements on interactivity to C. I did check with Walter and he confirmed it wasn't acceptable from his stance either.

D has fibers with which you can implement a thread model similar to the one in Go based on channels with blocking takes using green threads.

Actually, in project Loom carried out by Oracle the respective changeds wre made in JVM to bring fibers to the JVM. With the use of fibers some thread model was implemented that is quite close to the thread model in Go.

The work developed in project Loom has been released on the JVM since by Oracle since at least JDK 19.

So, something like communicating sequential processes (CS) as propsed by Tony Hoare and implemented by the people at Go into Go could also be brought to D. I think this would make D very interesting for sever-side development. Aside from server-side development, with CSP concurrent programming in general becomes a lot easier and less painful with fewer deadlocks and race cnditions from the beginning. When they still occur they are much easier to reproduce and understand and to fix them.

On 08/11/2024 4:31 AM, Bienlein wrote:
> On Tuesday, 6 August 2024 at 02:00:39 UTC, Richard (Rikki) Andrew Cattermole wrote:
>> On 06/08/2024 6:11 AM, Sebastiaan Koppe wrote:
>>> It is similar to the approach Go takes in that it makes calling C functions more difficult though.
>>
>> Go goes a bit further than that, their goroutines are effectively in their own calling convention.
>>
>> We cannot replicate what they did, as we have strict requirements on interactivity to C. I did check with Walter and he confirmed it wasn't acceptable from his stance either.
> 
> D has fibers with which you can implement a thread model similar to the one in Go based on channels with blocking takes using green threads.

They are both stackful coroutines yes, but they are implemented completely differently.

D's fibers cannot scale.

On Thursday, 7 November 2024 at 16:03:45 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 08/11/2024 4:31 AM, Bienlein wrote:
>> On Tuesday, 6 August 2024 at 02:00:39 UTC, Richard (Rikki) Andrew Cattermole wrote:
>>> On 06/08/2024 6:11 AM, Sebastiaan Koppe wrote:
>>>> It is similar to the approach Go takes in that it makes calling C functions more difficult though.
>>>
>>> Go goes a bit further than that, their goroutines are effectively in their own calling convention.
>>>
>>> We cannot replicate what they did, as we have strict requirements on interactivity to C. I did check with Walter and he confirmed it wasn't acceptable from his stance either.
>> 
>> D has fibers with which you can implement a thread model similar to the one in Go based on channels with blocking takes using green threads.
>
> They are both stackful coroutines yes, but they are implemented completely differently.
>
> D's fibers cannot scale.

Okay, I see. Thanks for telling me. What a pitty, would be a real bone for D.

On 08/11/2024 5:20 AM, Bienlein wrote:
> On Thursday, 7 November 2024 at 16:03:45 UTC, Richard (Rikki) Andrew Cattermole wrote:
>> D's fibers cannot scale.
> 
> Okay, I see. Thanks for telling me. What a pitty, would be a real bone for D.

This is one of the reasons we need stackless coroutines, they scale.