Jump to page: 1 2
Thread overview
August 05

Coroutines is a stack machine transformed representation for a function. It enables storing the state of a function externally to the stack to allow for high throughput event handling.

Latest: https://gist.github.com/rikkimax/fe2578e1dfbf66346201fd191db4bdd4

Current: https://gist.github.com/rikkimax/fe2578e1dfbf66346201fd191db4bdd4/0bb4442c092e061d520c35a43278f0819666f26f

It uses the @async attribute to transform into a type that is used to describe the function, with the help of implicit conversions using knowable library types and a hook for construction to produce a library object that represents the function.

There is support for such UDA's as @Route to make the function automatically marked as @async to prevent explicit annotation requirement.

An example usage of one:

void clientCO(Socket socket) @async {
	writeln("Connection has been made");

	socket.write("GET / HTTP/1.1\r\n");
	socket.write("Accept-Encoding: identity\r\n");
	socket.write("\r\n");

	while(Future!string readLine = socket.readUntil("\n")) {
		if (!readLine.isComplete) {
			writeln("Not alive and did not get a result");
			return;
		}
		
		string result = readLine.result;
		writeln(result);

		if (result == "</html>") {
			writeln("Saw end of expected input");
			return;
		}
	}
}
August 05

On Monday, 5 August 2024 at 01:36:31 UTC, Richard (Rikki) Andrew Cattermole wrote:

>

Coroutines is a stack machine transformed representation for a function. It enables storing the state of a function externally to the stack to allow for high throughput event handling.

Latest: https://gist.github.com/rikkimax/fe2578e1dfbf66346201fd191db4bdd4

Current: https://gist.github.com/rikkimax/fe2578e1dfbf66346201fd191db4bdd4/0bb4442c092e061d520c35a43278f0819666f26f

It uses the @async attribute to transform into a type that is used to describe the function, with the help of implicit conversions using knowable library types and a hook for construction to produce a library object that represents the function.

There is support for such UDA's as @Route to make the function automatically marked as @async to prevent explicit annotation requirement.

An example usage of one:

void clientCO(Socket socket) @async {
	writeln("Connection has been made");

	socket.write("GET / HTTP/1.1\r\n");
	socket.write("Accept-Encoding: identity\r\n");
	socket.write("\r\n");

	while(Future!string readLine = socket.readUntil("\n")) {
		if (!readLine.isComplete) {
			writeln("Not alive and did not get a result");
			return;
		}
		
		string result = readLine.result;
		writeln(result);

		if (result == "</html>") {
			writeln("Saw end of expected input");
			return;
		}
	}
}

C#'s async model is not an example to follow, your Future!T is the proof

There should be no distinction between an async/sync function

August 05
On 05/08/2024 10:17 PM, ryuukk_ wrote:
> C#'s async model is not an example to follow, your |Future!T| is the proof
> 
> There should be no distinction between an async/sync function

I started with the position that the compiler should be able to do it completely transparently to the user. It failed. Too many holes in it. It was not implementable at all.

This is why I had to remove synchronous function support out of the DIP, because it would do the wrong thing guaranteed.

Also the ``Future`` type in the DIP is a library type, the DIP itself does not introduce it. But it is something I know works, as I have it implemented already.
August 05

On Monday, 5 August 2024 at 01:36:31 UTC, Richard (Rikki) Andrew Cattermole wrote:

>

Coroutines is a stack machine transformed representation for a function. It enables storing the state of a function externally to the stack to allow for high throughput event handling.

Thank you for spearheading this. Having coroutines in the language would be a big step forward, and hopefully pave the way for more non-blocking programming in D.

There are a few points I would like to bring up early though.

  • I see no mention of C++'s coroutines. I think it would be good to learn from their design and implementation.

  • As you might know I am a big proponent of C++'s Senders/Receivers (a.k.a. P2300). One awesome integration they have is that Senders can be awaited by coroutines, and coroutines can await Senders. This allows for users to pick their preference, e.g. use convenient coroutines but suffer some allocation costs, and use Senders for more performant sections if need be. See
    https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2300r7.html#design-awaitables-are-senders and https://ericniebler.com/2024/02/04/what-are-senders-good-for-anyway/

  • I don't understand the choice for @async. Sometimes resembling the word "coroutine" would be better in my opinion.

  • In while(Future!string readLine = socket.readUntil("\n")) I suspect it to do an explicit await. How does that work?

  • I find the use of Future a bit confusing. Futures generally carry too much synchronisation overhead with them and I doubt coroutines need a full Future implementation anyway. Perhaps call it a Task like they do in C++?

  • The use of @isasync is a bit too cute for me. It also hides the fact sometime is a coroutine.

  • Instead of allowing a coroutine in a function that is not a coroutine itself - and injecting a blocking call - I would require explicit call to evaluate the coroutine instead. No magic.

  • Can coroutines be continued on another thread, assuming sequential consistency?

August 05
On Monday, 5 August 2024 at 10:30:24 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 05/08/2024 10:17 PM, ryuukk_ wrote:
>> C#'s async model is not an example to follow, your |Future!T| is the proof
>> 
>> There should be no distinction between an async/sync function
>
> I started with the position that the compiler should be able to do it completely transparently to the user. It failed. Too many holes in it. It was not implementable at all.

The only way it can work is to make all functions async and have the compiler do an optimisation pass to turn those it can proof into sync ones.

It is similar to the approach Go takes in that it makes calling C functions more difficult though.

August 06
On 06/08/2024 6:11 AM, Sebastiaan Koppe wrote:
> It is similar to the approach Go takes in that it makes calling C functions more difficult though.

Go goes a bit further than that, their goroutines are effectively in their own calling convention.

We cannot replicate what they did, as we have strict requirements on interactivity to C. I did check with Walter and he confirmed it wasn't acceptable from his stance either.
August 06
On 06/08/2024 3:50 AM, Sebastiaan Koppe wrote:
> On Monday, 5 August 2024 at 01:36:31 UTC, Richard (Rikki) Andrew Cattermole wrote:
>> Coroutines is a stack machine transformed representation for a function. It enables storing the state of a function externally to the stack to allow for high throughput event handling.
> 
> Thank you for spearheading this. Having coroutines in the language would be a big step forward, and hopefully pave the way for more non-blocking programming in D.
> 
> There are a few points I would like to bring up early though.
> 
> - I see no mention of C++'s coroutines. I think it would be good to learn from their design and implementation.
>
> - As you might know I am a big proponent of C++'s Senders/Receivers (a.k.a. P2300). One awesome integration they have is that Senders can be awaited by coroutines, and coroutines can await Senders. This allows for users to pick their preference, e.g. use convenient coroutines but suffer some allocation costs, and use Senders for more performant sections if need be. See
> https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2300r7.html#design-awaitables-are-senders and https://ericniebler.com/2024/02/04/what-are-senders-good-for-anyway/

My conclusion about P2300 is that it is all library code. This DIP would allow you to implement it. Although some language integration may be missing as the data processing use case is not covered here (I have no experience with it in context and so currently I am not the best person for it).

> - I don't understand the choice for `@async`. Sometimes resembling the word "coroutine" would be better in my opinion.

It was ``@co`` for a long time.

I had a number of people request ``async`` instead.

> - In `while(Future!string readLine = socket.readUntil("\n"))` I suspect it to do an explicit await. How does that work?

When a coroutine is acquired, it'll yield automatically, it is implicit.

We may want to support explicit yielding via a multiple return.

I should probably add this, however I'm divided upon it currently. Need time to think about it some more.

> - I find the use of `Future` a bit confusing. Futures generally carry too much synchronisation overhead with them and I doubt coroutines need a full Future implementation anyway. Perhaps call it a Task like they do in C++?

This DIP does not propose library code. You are free to write whatever library code you want and have it construct itself from the language representation of a coroutine.

This is a key design goal as there is a good chance we'll want different solutions for different tasks. Being stuck with only one solution (like how new'ing a class calls a specific hook) will only lead to people being unhappy.

Note: we can introduce hooking into a specific library with a new expression, however that is not present in this DIP as we have enough to get on with with just this DIP.

> - The use of `@isasync` is a bit too cute for me. It also hides the fact sometime is a coroutine.

I want to be placing a lot more emphasis on UDA's like ``@Route``, what we do right now with reflection is far too costly in terms of how to register symbols.

But yes, I do want to hide that it is a coroutine. The average person shouldn't have to care if its asynchronous of synchronous, it does not enable them to archive business goals faster.

> - Instead of allowing a coroutine in a function that is not a coroutine itself - and injecting a blocking call - I would require explicit call to evaluate the coroutine instead. No magic.

And that is library code ;)

But yes, I did try to explain that you did not need language integration for this. In fact the prime sieve example show cases exactly what you suggest!

> - Can coroutines be continued on another thread, assuming sequential consistency?

Yes. There is nothing preventing it. There is also nothing making it safe either.
August 08

On Monday, 5 August 2024 at 01:36:31 UTC, Richard (Rikki) Andrew Cattermole wrote:

I see you linked to that classic article on function colouring, but it wasn't clear to me how your DIP avoids the function colouring problem. We'd basically need something like parametric polymorphism over function effects, unless there's some trick or assumptions we can make to circumvent it.

August 08
On 08/08/2024 1:37 PM, Meta wrote:
> On Monday, 5 August 2024 at 01:36:31 UTC, Richard (Rikki) Andrew Cattermole wrote:
> 
> I see you linked to that classic article on function colouring, but it wasn't clear to me how your DIP avoids the function colouring problem. We'd basically need something like parametric polymorphism over function effects, unless there's some trick or assumptions we can make to circumvent it.

Originally there was meant to be behavior in synchronous functions to add blocking automatically.

That way it didn't matter how you "yield" implicitly.

Unfortunately that had major problems, and had to be removed.
September 22

On Monday, 5 August 2024 at 01:36:31 UTC, Richard (Rikki) Andrew Cattermole wrote:

>

Coroutines is a stack machine transformed representation for a function. It enables storing the state of a function externally to the stack to allow for high throughput event handling.

[...]

This deserves more attention

« First   ‹ Prev
1 2