On 29/03/2025 12:41 AM, Dukc wrote:
>     Also many approaches are expecting async/await behavior
> 
> Is this a problem for D? Doesn't Vibe.D, or even just Phobos, handle this just fine?

No.

D's Fiber's cannot move between threads safely, and there is a limit on the number you can have at the kernel level due to the usage of pages.

We cannot recommend a solution to drive modern event loops (circa late 90's), due to safety concerns and these limitations. Note the word recommend, it is important here!

Hence why we need stackless coroutines to fill in that gap ;)

On Friday, 28 March 2025 at 14:15:01 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 29/03/2025 12:41 AM, Dukc wrote:
>>     Also many approaches are expecting async/await behavior
>> 
>> Is this a problem for D? Doesn't Vibe.D, or even just Phobos, handle this just fine?
>
> No.
>
> D's Fiber's cannot move between threads safely, and there is a limit on the number you can have at the kernel level due to the usage of pages.
>
> We cannot recommend a solution to drive modern event loops (circa late 90's), due to safety concerns and these limitations. Note the word recommend, it is important here!
>
> Hence why we need stackless coroutines to fill in that gap ;)

Actually, one could have light weight stackful coroutines within the system, and user space switching.  Have a look at what Alef had - the predecessor to Limbo and Go.

It had 'procs' and 'tasks'.  The former being implemented simply as kernel level threads, hence preemptively scheduled by the kernel, the latter as user level coroutines within such a thread.  The same scheme ended up as libthread for C on plan9.

One should be able to create a libthread like facility for D, and given the GC it should be almost as usable as the CSP scheme in Go.  The difference being the lack of preemption between 'tasks', vs the preemption which happens with goroutines.

As to stackless coroutines, I personally find the too much hassle, hiding some (but not all) of the detials and complexities of their underlying state-machine implementation.

About the only thing I'd say they have going for them is they sort of are flavour of the month, and C++ has sort of done the same thing.  But then I'm biased, I prefer the CSP / Actor approach.

On 29/03/2025 5:35 AM, Derek Fawcus wrote:
> On Friday, 28 March 2025 at 14:15:01 UTC, Richard (Rikki) Andrew Cattermole wrote:
>> On 29/03/2025 12:41 AM, Dukc wrote:
>>>     Also many approaches are expecting async/await behavior
>>>
>>> Is this a problem for D? Doesn't Vibe.D, or even just Phobos, handle this just fine?
>>
>> No.
>>
>> D's Fiber's cannot move between threads safely, and there is a limit on the number you can have at the kernel level due to the usage of pages.
>>
>> We cannot recommend a solution to drive modern event loops (circa late 90's), due to safety concerns and these limitations. Note the word recommend, it is important here!
>>
>> Hence why we need stackless coroutines to fill in that gap ;)
> 
> Actually, one could have light weight stackful coroutines within the system, and user space switching.  Have a look at what Alef had - the predecessor to Limbo and Go.
>
> It had 'procs' and 'tasks'.  The former being implemented simply as kernel level threads, hence preemptively scheduled by the kernel, the latter as user level coroutines within such a thread.  The same scheme ended up as libthread for C on plan9.

We have that, its called a Fiber.

> One should be able to create a libthread like facility for D, and given the GC it should be almost as usable as the CSP scheme in Go.  The difference being the lack of preemption between 'tasks', vs the preemption which happens with goroutines.

A goroutine effectively has its own calling convention. That cannot call out to C.

We have dismissed that as an option.

> As to stackless coroutines, I personally find the too much hassle, hiding some (but not all) of the detials and complexities of their underlying state-machine implementation.

Whatever solution we have that is to be recommended it must be thread safe, can be moved between threads and can have a lot of them (like over 100K/s). Oh and must also be able to call C.

The only solution that matches that, and is known to be a mature concept in use with these requirements is stackless coroutines.

Everything else fails some part of the requirements.

> About the only thing I'd say they have going for them is they sort of are flavour of the month, and C++ has sort of done the same thing.  But then I'm biased, I prefer the CSP / Actor approach.

I'm exploring actors as the event handling abstraction for windowing.

However you'd still use coroutines as part of this. They are complimentary.

On Friday, 28 March 2025 at 17:04:58 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 29/03/2025 5:35 AM, Derek Fawcus wrote:
>> On Friday, 28 March 2025 at 14:15:01 UTC, Richard (Rikki) Andrew Cattermole wrote:
>>> On 29/03/2025 12:41 AM, Dukc wrote:
>>>>     Also many approaches are expecting async/await behavior
>>>>
>>>> Is this a problem for D? Doesn't Vibe.D, or even just Phobos, handle this just fine?
>>>
>>> No.
>>>
>>> D's Fiber's cannot move between threads safely, and there is a limit on the number you can have at the kernel level due to the usage of pages.
>>>
>>> We cannot recommend a solution to drive modern event loops (circa late 90's), due to safety concerns and these limitations. Note the word recommend, it is important here!
>>>
>>> Hence why we need stackless coroutines to fill in that gap ;)
>> 
>> Actually, one could have light weight stackful coroutines within the system, and user space switching.  Have a look at what Alef had - the predecessor to Limbo and Go.
> >
>> It had 'procs' and 'tasks'.  The former being implemented simply as kernel level threads, hence preemptively scheduled by the kernel, the latter as user level coroutines within such a thread.  The same scheme ended up as libthread for C on plan9.
>
> We have that, its called a Fiber.

The fiber is only part of the answer.  My point in raising Alef was to show that one doesn't actually need migration between threads, an alternative is viable. It rather depends upon how one wishes to handle i/o.

>
>> One should be able to create a libthread like facility for D, and given the GC it should be almost as usable as the CSP scheme in Go.  The difference being the lack of preemption between 'tasks', vs the preemption which happens with goroutines.
>
> A goroutine effectively has its own calling convention. That cannot call out to C.

Nope.  That is the goroutine as implemented by the Google gc compiler.  The goroutines as implemented by GCC can call out to C (they run as threads).

The two concepts are orthogonal, it is the fact that gc goroutines have an automatic user space thread scheduler and its minimal initial stack which makes calling to C awkward, not the calling convention per-se.


Now my prefered form would be something like the Alef scheme, but is an optional user space scheduler.  Then one can easily have 3 forms of concurrent stackful routines. However I've yet to try coding that up, and so don't have anything to point to.

On Friday, 28 March 2025 at 17:04:58 UTC, Richard (Rikki) Andrew Cattermole wrote:
>>>
>>> Hence why we need stackless coroutines to fill in that gap ;)

BTW - having them as a library facility would not bother me, I'd just not use them.

So if you desire their creation, knock yourself out! :-)

On 29/03/2025 6:28 AM, Derek Fawcus wrote:
> On Friday, 28 March 2025 at 17:04:58 UTC, Richard (Rikki) Andrew Cattermole wrote:
>> On 29/03/2025 5:35 AM, Derek Fawcus wrote:
>>> On Friday, 28 March 2025 at 14:15:01 UTC, Richard (Rikki) Andrew Cattermole wrote:
>>>> On 29/03/2025 12:41 AM, Dukc wrote:
>>>>>     Also many approaches are expecting async/await behavior
>>>>>
>>>>> Is this a problem for D? Doesn't Vibe.D, or even just Phobos, handle this just fine?
>>>>
>>>> No.
>>>>
>>>> D's Fiber's cannot move between threads safely, and there is a limit on the number you can have at the kernel level due to the usage of pages.
>>>>
>>>> We cannot recommend a solution to drive modern event loops (circa late 90's), due to safety concerns and these limitations. Note the word recommend, it is important here!
>>>>
>>>> Hence why we need stackless coroutines to fill in that gap ;)
>>>
>>> Actually, one could have light weight stackful coroutines within the system, and user space switching.  Have a look at what Alef had - the predecessor to Limbo and Go.
>> >
>>> It had 'procs' and 'tasks'.  The former being implemented simply as kernel level threads, hence preemptively scheduled by the kernel, the latter as user level coroutines within such a thread.  The same scheme ended up as libthread for C on plan9.
>>
>> We have that, its called a Fiber.
> 
> The fiber is only part of the answer.  My point in raising Alef was to show that one doesn't actually need migration between threads, an alternative is viable. It rather depends upon how one wishes to handle i/o.

You need that migration to take advantage of the advanced event loops like IOCP.

They utilize their own thread pool where work is given to whatever thread is available, no preference.

Otherwise you're stuck doing what vibe.d does. If you care about safety of course.

>>> One should be able to create a libthread like facility for D, and given the GC it should be almost as usable as the CSP scheme in Go.  The difference being the lack of preemption between 'tasks', vs the preemption which happens with goroutines.
>>
>> A goroutine effectively has its own calling convention. That cannot call out to C.
> 
> Nope.  That is the goroutine as implemented by the Google gc compiler.  The goroutines as implemented by GCC can call out to C (they run as threads).

"The gccgo compiler implements goroutines using a technique called segmented stacks, supported by recent modifications to the gold linker. Gollvm similarly is built on the corresponding LLVM infrastructure."

https://go.dev/doc/faq

Is this out of date?

Because that is still using the read barriers and is still implemented using a different calling convention that we have indeed eliminated.

> The two concepts are orthogonal, it is the fact that gc goroutines have an automatic user space thread scheduler and its minimal initial stack which makes calling to C awkward, not the calling convention per-se.

I've read the implementation, it'll error if you try.

On Friday, 28 March 2025 at 17:36:00 UTC, Richard (Rikki) Andrew Cattermole wrote:
>
> You need that migration to take advantage of the advanced event loops like IOCP.

Actually, I had in mind trying to do something based on linux io_uring, though that would obviously be Linux specific...

> "The gccgo compiler implements goroutines using a technique called segmented stacks, supported by recent modifications to the gold linker. Gollvm similarly is built on the corresponding LLVM infrastructure."
>
> https://go.dev/doc/faq
>
> Is this out of date?

Apparently...

https://gcc.gnu.org/onlinedocs/gccgo/

I've managed to run out of memory with gccgo, simply because each created goroutine uses a native thread.

I've manually called C routines from gccgo directly, without using cgo, making use of the techniques in the above manual, and had it call in to C very quickly, at native speed, and apparently on a native thread stack.

DF

On Thursday, 27 March 2025 at 14:21:11 UTC, Ali Çehreli wrote:
> I haven't watched it yet. I'm just attempting to beat you to announce it. :p
>
>   https://www.youtube.com/watch?v=_kRsbu82zqs
>
> Ali

I was wondering. At around 1:20:00 he is experimenting with opApply() to make his Array foreach-able. He makes an obvious error of shadowing i parameter with the loop index i. I was wondering why the compiler didn't catch this shadowing as it does in general do recognize shadowed variables.

On 3/28/2025 1:51 AM, Ogion wrote:
> Is there a good reason why `main` isn’t `extern(C)` by default in BetterC?

Mainly having the compiler behave consistently.