On Apr 20, 2012, at 10:21 AM, Andrew Lauritzen wrote:

>> To be precise, fibers in and of themselves aren't exacty what I want, but are the best means to getting it that I've seen so far. They enable efficient implementation of coroutines, which many languages either cannot express at all, or can only express very poorly by using the sledgehammer of a full-on kernel thread to get it. A call stack is a very useful way to group logic, and being forced to go outside the language and ask the OS for another one is a shame.
>> 
>> Game logic is an area where this technique REALLY shines. Case in point: Unity3D. The entire engine is built around C#'s iterator method facility to implement coroutines to control entity logic. Hundreds or even thousands of them can be active with very little overhead compared to a full thread context switch. Unfortunately, they're hamstrung by not being a true coroutine (you can only yield from the top frame).
>> 
> Right, exactly the above! Fibers are totally uninteresting as a "lighter thread" or unit of scheduling for the reasons that you note, but coroutines are just a better way to write a lot of code. This is basically the entire premise for the Go programming language, so it's worth taking a peek at that if you haven't already.

Okay, here's the talk:

http://petermodzelewski.blogspot.com/2008/10/tango-conference-2008-fibers-talk-video.html

I've been digging around and just found Mikola's coroutine implementation on my hard drive.  It would have to be updated, but I do have the code.  Maybe this should be added to Druntime.


>> Stack overflow?  Give the fiber a larger stack when you create it. The default is really rather small.
> 
> I'm fairly certain that it's not a "real" stack overflow... the program continues to operate normally unless the debugger is stepping through and it only happens when an exception is thrown. And it happens pretty much always when an exception is thrown, you just won't see it unless you have a debugger attached to see the output. So like I said, it is somewhat worrisome, but the program seems to be running properly despite it, so it may be a red herring.

It could be a "real" stack overflow.  I can't remember the details, but this happened once before when the default stack size was 4k.  Something about how exceptions are thrown right now has pretty considerable stack requirements on Windows--maybe the stack trace generation?  Anyway, that's why the default stack size for fibers is currently 8k IIRC.

On Apr 20, 2012, at 10:59 AM, Andrew Lauritzen wrote:

> On Friday, 20 April 2012 at 17:49:52 UTC, Sean Kelly wrote:
>> There wouldn't me much of a performance hit, mostly an additional allocation and a bitcopy when creating a Fiber.  It's more that making this work on platforms with built-in TLS could be quite tricky.
> Note that this would somewhat sabotage their usefulness as coroutines, depending on how it was implemented. That's not to say the idea isn't good (but I'd frame it more like "tasks"; see Thread Building Blocks or similar), but fibers/coroutines as they stand now are useful so I'd hate to see that capability lost.

Ah, coroutines would actually want to share TLS with the executing thread?  That's good to know.

On Friday, 20 April 2012 at 18:09:37 UTC, Sean Kelly wrote:
> Ah, coroutines would actually want to share TLS with the executing thread?  That's good to know.
In my case that's definitely useful as, like I said, I'm not trying to express any parallelism here and don't really want to take on the burden of synchronizing the data structures :) I'm merely using coroutines as a handy way to save and restore state, the alternative being to do it all manually at every yield point (yikes!).

On Friday, 20 April 2012 at 18:08:48 UTC, Sean Kelly wrote:
> It could be a "real" stack overflow.  I can't remember the details, but this happened once before when the default stack size was 4k.  Something about how exceptions are thrown right now has pretty considerable stack requirements on Windows--maybe the stack trace generation?  Anyway, that's why the default stack size for fibers is currently 8k IIRC.
Interesting, ok I'll play with the stack size a bit and see what happens :)

On Friday, 20 April 2012 at 18:09:37 UTC, Sean Kelly wrote:
> On Apr 20, 2012, at 10:59 AM, Andrew Lauritzen wrote:
>
>> On Friday, 20 April 2012 at 17:49:52 UTC, Sean Kelly wrote:
>>> There wouldn't me much of a performance hit, mostly an additional allocation and a bitcopy when creating a Fiber.  It's more that making this work on platforms with built-in TLS could be quite tricky.
>> Note that this would somewhat sabotage their usefulness as coroutines, depending on how it was implemented. That's not to say the idea isn't good (but I'd frame it more like "tasks"; see Thread Building Blocks or similar), but fibers/coroutines as they stand now are useful so I'd hate to see that capability lost.
>
> Ah, coroutines would actually want to share TLS with the executing thread?  That's good to know.

Yeah, I think there's a subtlety at play here, but it's an important one. In most languages and environments, a thread of execution and a call stack are conflated into a single entity. I think what Andrew and I are driving at is that we'd like them to be orthogonal concepts.

A "thread" is merely a lane of execution, and that lane can play host to any call-stack "fiber" context we'd like it to. For this reason, it's important that fibers not be bound to the thread that initially created them. Mono implemented a continuation system, but the stack context is tied to the thread that spawned it, which is frustrating and prevents efficient task pooling.

On Apr 20, 2012, at 1:42 PM, Jameson Ernst wrote:

> On Friday, 20 April 2012 at 18:09:37 UTC, Sean Kelly wrote:
>> On Apr 20, 2012, at 10:59 AM, Andrew Lauritzen wrote:
>> 
>>> On Friday, 20 April 2012 at 17:49:52 UTC, Sean Kelly wrote:
>>>> There wouldn't me much of a performance hit, mostly an additional allocation and a bitcopy when creating a Fiber.  It's more that making this work on platforms with built-in TLS could be quite tricky.
>>> Note that this would somewhat sabotage their usefulness as coroutines, depending on how it was implemented. That's not to say the idea isn't good (but I'd frame it more like "tasks"; see Thread Building Blocks or similar), but fibers/coroutines as they stand now are useful so I'd hate to see that capability lost.
>> 
>> Ah, coroutines would actually want to share TLS with the executing thread?  That's good to know.
> 
> Yeah, I think there's a subtlety at play here, but it's an important one. In most languages and environments, a thread of execution and a call stack are conflated into a single entity. I think what Andrew and I are driving at is that we'd like them to be orthogonal concepts.
> 
> A "thread" is merely a lane of execution, and that lane can play host to any call-stack "fiber" context we'd like it to. For this reason, it's important that fibers not be bound to the thread that initially created them. Mono implemented a continuation system, but the stack context is tied to the thread that spawned it, which is frustrating and prevents efficient task pooling.

The weird thing in D is that static data is thread-local by default, so moving a fiber between threads would mean the value of statics would change between calls.  I had thought that by giving each fiber its own TLS space the programming model would be more predictable, but I can see how the current behavior may actually be desirable in some instances.  I wonder if there's any value in having a child of Fiber that has its own TLS, call it CoopThread or whatever.

For the record, there's absolutely nothing preventing you from moving a fiber across threads in Druntime.  It's absolutely legal behavior, despite the potential weirdness with statics.

On Friday, 20 April 2012 at 18:08:48 UTC, Sean Kelly wrote:
> Okay, here's the talk:
>
> http://petermodzelewski.blogspot.com/2008/10/tango-conference-2008-fibers-talk-video.html

Finally got a chance to read through the slides here - great stuff! This describes precisely what we're talking about and even gives examples of how the ugly state machine code works. It even calls out both games and network servers as examples of where the pattern is extremely useful :)

On 18 April 2012 21:48, Andrew Lauritzen <andrew.lauritzen@gmail.com> wrote:

> I sent this to the mailing list but as it appears to be awaiting moderation, here might be a more appropriate place for discussion.
>
> Throwing exceptions from fibers seems buggy on Windows in the latest D2 runtime. Even a small modification to the example given in the unit test (adding unrelated exception handling to the same scope) fails with a "Stack Overflow" exception then tons of access violations:
>
>   enum MSG = "Test message.";
>   string caughtMsg;
>   (new Fiber({
>       try
>       {
>           throw new Exception(MSG);
>       }
>       catch (Exception e)
>       {
>           caughtMsg = e.msg;
>       }
>   })).call();
>   assert(caughtMsg == MSG);
>   // Add these two lines
>   try { caughtMsg = "Hello"; }
>   catch (Exception e) { caughtMsg = "World"; }
>
> It seems brittle as well. Adding a "assert(caughtMsg == "Hello");" to the end of the above program for instance avoids the access violations (but still shows the stack overflow in the debugger). Similarly playing with the caughtMsg assignments (omitting one) or adding another assert at the end of the program will sometimes cause the program to run without error. This is in debug mode with optimizations off to theoretically avoid dead code elimination, but clearly something is easily thrown off.
>
> I take it that not too many people are using Fibers on windows, but I really need them for a project that I'm working on. Alternatively I've considered trying to wrap native Win32 API Fibers to accomplish something similar, but I'm not sure of the difficulty of that. Anyone done this or have recommendations of a similar workaround? I really do need both fibers/coroutines and exceptions to express what I'm trying to do in a clean and efficient manner. I've considered switching to Go, but I've got some really handy mixins in D that would become copy-pasted stuff in Go presumably.
>
> I'm running a 32-bit D application on Windows 7 x64. I've tried on several machines with the same results, and the same issue was present in 2.058 as well.
>
> Any ideas?
>

How far did you get using Fibers on Windows in the end?
I'm using GDC for x64 Windows, and it crashes in call(), but before it
enters the fibre function. I think it's just broken... :/
Anyone else had problems?

On Monday, 21 May 2012 at 15:04:56 UTC, Manu wrote:
> How far did you get using Fibers on Windows in the end?
> I'm using GDC for x64 Windows, and it crashes in call(), but before it
> enters the fibre function. I think it's just broken... :/
> Anyone else had problems?

Fiber support should be reasonably stable on pretty much all flavors of Windows when using DMD (i.e. compiling for 32 bit). Never tested GDC…

David

On 21 May 2012 19:16, David Nadlinger <see@klickverbot.at> wrote:

> On Monday, 21 May 2012 at 15:04:56 UTC, Manu wrote:
>
>> How far did you get using Fibers on Windows in the end?
>> I'm using GDC for x64 Windows, and it crashes in call(), but before it
>> enters the fibre function. I think it's just broken... :/
>> Anyone else had problems?
>>
>
> Fiber support should be reasonably stable on pretty much all flavors of Windows when using DMD (i.e. compiling for 32 bit). Never tested GDC…


Hmm, well I am limited to x64, and therefore GDC... :/