On Friday, 15 August 2014 at 14:26:28 UTC, Sean Kelly wrote:
> Oh handy, so there's basically no work to be done on Windows.  I'll have to check the behavior of mmap on Posix.

I heard, calloc behaves this way on linux (COW blank page mapped to the entire range), it was discussed here some time ago.

On Friday, 15 August 2014 at 14:28:34 UTC, Dicebot wrote:
> Won't that kind of kill the purpose of Fiber as low-cost context abstraction? Stack size does add up for thousands of fibers.

I didn't measure it.

On Friday, 15 August 2014 at 14:28:34 UTC, Kagamin wrote:
> On Friday, 15 August 2014 at 14:26:28 UTC, Sean Kelly wrote:
>> Oh handy, so there's basically no work to be done on Windows.  I'll have to check the behavior of mmap on Posix.
>
> I heard, calloc behaves this way on linux (COW blank page mapped to the entire range), it was discussed here some time ago.

Another abstraction for core.vmm.

On Friday, 15 August 2014 at 14:26:28 UTC, Sean Kelly wrote:
> On Friday, 15 August 2014 at 08:36:34 UTC, Kagamin wrote:
>> http://msdn.microsoft.com/en-us/library/windows/desktop/aa366887%28v=vs.85%29.aspx
>>> Allocates memory charges (from the overall size of memory and the paging files on disk) for the specified reserved memory pages. The function also guarantees that when the caller later initially accesses the memory, the contents will be zero. Actual physical pages are not allocated unless/until the virtual addresses are actually accessed.
>
> Oh handy, so there's basically no work to be done on Windows.  I'll have to check the behavior of mmap on Posix.

It sounds like mmap (typically) works the same way on Linux, and that malloc generally does as well.  I'll have to test this to be sure.  If so, and if doing so is fast, I'm going to increase the default stack size on 64-bit systems to something reasonably large.  And here I thought I would have to do this all manually.

On Friday, 15 August 2014 at 14:28:34 UTC, Dicebot wrote:
>
> Won't that kind of kill the purpose of Fiber as low-cost context abstraction? Stack size does add up for thousands of fibers.

As long as allocation speed is fast for large allocs (which I have to test), I want to change the default size to be very large.  The virtual address space in a 64-bit app is enormous, and if the memory is committed on demand then physical memory use should only match what the user actually requires.  This should allow us to create millions of fibers and not overrun system memory, and also not worry about stack overruns, which has always been a concern with the default fiber stack size.

At least on OSX, it appears that mapping memory is constant time regardless of size, but there is some max total memory I'm allowed to map, presumably based on the size of a vmm lookup tabe.  The max block size I can allocate is 1 GB, and I can allocate roughly 131,000 of these blocks before getting an out of memory error.  If I reduce the block size to 4 MB I can allocate more than 10M blocks without error.  I think some default stack size around 4 MB seems about right.  Increasing the size to 16 MB failed after about 800,000 allocations, which isn't enough (potential) fibers.

On Friday, 15 August 2014 at 14:45:02 UTC, Sean Kelly wrote:
> On Friday, 15 August 2014 at 14:28:34 UTC, Dicebot wrote:
>>
>> Won't that kind of kill the purpose of Fiber as low-cost context abstraction? Stack size does add up for thousands of fibers.
>
> As long as allocation speed is fast for large allocs (which I have to test), I want to change the default size to be very large.  The virtual address space in a 64-bit app is enormous, and if the memory is committed on demand then physical memory use should only match what the user actually requires.  This should allow us to create millions of fibers and not overrun system memory, and also not worry about stack overruns, which has always been a concern with the default fiber stack size.

No, I was referring to the proposal to supply bigger stack size to Fiber constructor - AFAIR it currently does allocate that memory eagerly (and does not use any OS CoW tools), doesn't it?

On Friday, 15 August 2014 at 15:25:23 UTC, Dicebot wrote:
>
> No, I was referring to the proposal to supply bigger stack size to Fiber constructor - AFAIR it currently does allocate that memory eagerly (and does not use any OS CoW tools), doesn't it?

I thought it did, but apparently the behavior of VirtualAlloc and mmap (which Fiber uses to allocate the stack) simply reserves the range and then commits it lazily, even though what you've told it to do is allocate the memory.  This is really great news since it means that no code changes will be required to do the thing I wanted to do anyway.

On Friday, 15 August 2014 at 15:40:35 UTC, Sean Kelly wrote:
> On Friday, 15 August 2014 at 15:25:23 UTC, Dicebot wrote:
>>
>> No, I was referring to the proposal to supply bigger stack size to Fiber constructor - AFAIR it currently does allocate that memory eagerly (and does not use any OS CoW tools), doesn't it?
>
> I thought it did, but apparently the behavior of VirtualAlloc and mmap (which Fiber uses to allocate the stack) simply reserves the range and then commits it lazily, even though what you've told it to do is allocate the memory.  This is really great news since it means that no code changes will be required to do the thing I wanted to do anyway.

Guess that means some experimenting this weekend for me too :)

On Thursday, 14 August 2014 at 18:52:00 UTC, Sean Kelly wrote:
> On 64 bit, reserve a huge chunk of memory, set a SEGV handler and commit more as needed. Basically how kernel thread stacks work. I've been meaning to do this but haven't gotten around to it yet.

Very nice; the hardware VM machinery takes care of it and there's only overhead when overflow occurs. D's Fibers will really be something remarkable for user-space with this facility.