On Friday, 15 August 2014 at 08:41:30 UTC, Kagamin wrote:
> On Thursday, 14 August 2014 at 07:46:29 UTC, Carl Sturtivant wrote:
>> The default size of the runtime stack for a Fiber is 4*PAGESIZE which is very small, and a quick test shows that a Fiber suffers a stack overflow that doesn't lead to a clean termination when this limit is exceeded.
>
> Pass a bigger stack size to the Fiber constructor?

No good if the stack size needed depends dynamically on the computation in that Fiber.

On Friday, 15 August 2014 at 15:40:35 UTC, Sean Kelly wrote:
> On Friday, 15 August 2014 at 15:25:23 UTC, Dicebot wrote:
>>
>> No, I was referring to the proposal to supply bigger stack size to Fiber constructor - AFAIR it currently does allocate that memory eagerly (and does not use any OS CoW tools), doesn't it?
>
> I thought it did, but apparently the behavior of VirtualAlloc and mmap (which Fiber uses to allocate the stack) simply reserves the range and then commits it lazily, even though what you've told it to do is allocate the memory.  This is really great news since it means that no code changes will be required to do the thing I wanted to do anyway.

Just read this after posting earlier replies! Very exciting.

I'll be doing some experiments to see how this works out.

What about at 32-bits?

On Friday, 15 August 2014 at 20:11:43 UTC, Carl Sturtivant wrote:
> On Friday, 15 August 2014 at 08:41:30 UTC, Kagamin wrote:
>> On Thursday, 14 August 2014 at 07:46:29 UTC, Carl Sturtivant wrote:
>>> The default size of the runtime stack for a Fiber is 4*PAGESIZE which is very small, and a quick test shows that a Fiber suffers a stack overflow that doesn't lead to a clean termination when this limit is exceeded.
>>
>> Pass a bigger stack size to the Fiber constructor?
>
> No good if the stack size needed depends dynamically on the computation in that Fiber.

Should have read further down the thread --- you're right as the memory is in effect merely reserved virtual memory and isn't actually allocated.

On Friday, 15 August 2014 at 20:17:51 UTC, Carl Sturtivant wrote:
> On Friday, 15 August 2014 at 15:40:35 UTC, Sean Kelly wrote:
>>
>> I thought it did, but apparently the behavior of VirtualAlloc and mmap (which Fiber uses to allocate the stack) simply reserves the range and then commits it lazily, even though what you've told it to do is allocate the memory.  This is really great news since it means that no code changes will be required to do the thing I wanted to do anyway.
>
> Just read this after posting earlier replies! Very exciting.
>
> I'll be doing some experiments to see how this works out.
>
> What about at 32-bits?

I'm sure it works the same, but reserving large chunks of memory
there would eat up the address space.  I think the default will
have to remain some reasonably low number on 32-bit.

On Fri, 15 Aug 2014 20:19:18 +0000
Carl Sturtivant via Digitalmars-d-learn
<digitalmars-d-learn@puremagic.com> wrote:

> Should have read further down the thread --- you're right as the memory is in effect merely reserved virtual memory and isn't actually allocated.
and we -- 32-bit addicts -- will run out of address space while 64-bit happy people will laugh looking at us. ;-)

On Friday, 15 August 2014 at 22:26:54 UTC, ketmar via Digitalmars-d-learn wrote:
> and we -- 32-bit addicts -- will run out of address space while 64-bit
> happy people will laugh looking at us. ;-)

You should only choose stack size carefully or keep data in TempAlloc instead of stack.

On Sat, 16 Aug 2014 17:49:28 +0000
Kagamin via Digitalmars-d-learn <digitalmars-d-learn@puremagic.com>
wrote:

> You should only choose stack size carefully or keep data in TempAlloc instead of stack.
but i can choose stack size carefully without such 'address reserving' feature. ;-)

On Friday, 15 August 2014 at 15:50:30 UTC, Dicebot wrote:
> On Friday, 15 August 2014 at 15:40:35 UTC, Sean Kelly wrote:
>> On Friday, 15 August 2014 at 15:25:23 UTC, Dicebot wrote:
>>>
>>> No, I was referring to the proposal to supply bigger stack size to Fiber constructor - AFAIR it currently does allocate that memory eagerly (and does not use any OS CoW tools), doesn't it?
>>
>> I thought it did, but apparently the behavior of VirtualAlloc and mmap (which Fiber uses to allocate the stack) simply reserves the range and then commits it lazily, even though what you've told it to do is allocate the memory.  This is really great news since it means that no code changes will be required to do the thing I wanted to do anyway.
>
> Guess that means some experimenting this weekend for me too :)

Looks like it indeed just magically works. I have used this simple program:

import core.thread;

void foo()
{
//  int[10240] array;
    Fiber.yield();
    for (;;) {}
}

void main()
{
    foreach (_ ; 0 .. 10_000)
    {
        //auto fiber = new Fiber(&foo);
        auto fiber = new Fiber(&foo, 100_000);
        fiber.call();
    }
}

And checked peak memory profile via `time --format="%M"`. Fiber constructor argument size did not change memory consumption at all, only changing size of fiber stack variables did.

> Am I missing something? Is there a clean and simple way to get Fiber to
> no longer suffer a stack overflow when implementing D-routines?

Simply choose a big enough stack size when creating your fibers.
http://dlang.org/library/core/thread/Fiber.this.html

It's fairly cheap to use a really big stack like 4MB, because memory pages are committed lazily by the OS.

On 08/15/2014 05:09 PM, Sean Kelly wrote:
> At least on OSX, it appears that mapping memory is constant time
> regardless of size, but there is some max total memory I'm allowed to
> map, presumably based on the size of a vmm lookup tabe.  The max block
> size I can allocate is 1 GB, and I can allocate roughly 131,000 of these
> blocks before getting an out of memory error.  If I reduce the block
> size to 4 MB I can allocate more than 10M blocks without error.  I think
> some default stack size around 4 MB seems about right.  Increasing the
> size to 16 MB failed after about 800,000 allocations, which isn't enough
> (potential) fibers.

While pages are committed lazily it's not free to map more memory.
For example the kernel will have to allocate and setup more page table entries.
Furthermore we might need to use MAP_NORESERVE to not run into overcommit limitations, but that's not available on all OSes I think.

It might make sense to raise the stack size again [1] if we had an idea what size would avoid common issues (vibe.d uses 64kB). 4MB is too much for the default size IMO.

Given that the stack size is configurable let's instead try to improve the diagnostic of Fiber stack overflows first, we should also map a guard page on posix and add a segfault handler to detect overflows.

[1]: https://github.com/D-Programming-Language/druntime/commit/9dca50fe65402fb3cdfbb689f1aca58dc835dce4#diff-8bb12ed976acf0a5132e877ec5a01ea8R3163