http://d.puremagic.com/issues/show_bug.cgi?id=1786

           Summary: Phobos cannot accommodate multiple stacks
           Product: D
           Version: unspecified
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: Phobos
        AssignedTo: bugzilla@digitalmars.com
        ReportedBy: default_357-line@yahoo.de


Since the current version of Phobos always scans one whole stack (defined as "stored stack bottom to ESP"), any advanced constructs which require multiple stacks (like coroutines) are completely impossible to implement without disabling the entire GC whenever a stackthread is somewhere active, which in multi-threaded programs might disable the GC completely.

This issue can be fixed by introducing a way to override the phobos-perceived stack top, thus limiting the GC to the actual stack. The attached patch against GDC 0.23's Phobos does this.

Another problem is that stacks should be allocated as they're being used; however, dynamic arrays cannot do that because resizing might require us to move the stack elsewhere, which causes problems if we're currently using it :)

A solution to this is using memory-mapped files to contain the stack (as they don't allocate memory unless it's being used, but only reserve some address space). This leads to another problem: if the entire MmFile stack is registered with the GC as a Range, the subsequent GC scan causes the operating system to allocate memory for the whole range of the stack, causing an explosion of memory usage.

The way to prevent this is to only register the specific range that is currently in use, After the threads have been stopped and ESPs updated, but before the garbage collection is run.

For this, a way is required for the program to register a callback with the GC that is evoked when the garbage collection is being run (this callback must of course not allocate any memory).

The attached patch adds this also.

The patch has been tested using the stackthreads implementation in scrapple.tools, which now depends on it. For all I know, it works.

Since it is impossible to solve the problems described above without this patch, I request it, or some functionally equivalent version, be included in the official Phobos trunk.

Looking forward to responses and criticism,
 --feep (downs).


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=1786





------- Comment #1 from default_357-line@yahoo.de  2008-01-14 14:06 -------
Created an attachment (id=218)
 --> (http://d.puremagic.com/issues/attachment.cgi?id=218&action=view)
Filename is descriptive enough.

Agh, forgot to attach the patch. Here is it.


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=1786


default_357-line@yahoo.de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
    Attachment #218|Filename is descriptive     |thread_add_stacktop_override
        description|enough.                     |_and_callback.patch




-- 

http://d.puremagic.com/issues/show_bug.cgi?id=1786


default_357-line@yahoo.de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |INVALID




------- Comment #2 from default_357-line@yahoo.de  2008-01-18 03:16 -------
Mikola Lysenko, the author of the Original StackThreads
(http://assertfalse.com/about.shtml), has pointed out a fascinating way of
making the GC work despite its shortcomings.
With this, I have to revise my original opinion - looks like making the GC work
with StackThreads is possible after all :)

Goes to show that using absolutes like "can't possibly work" is always a bad idea. Stupid me.

I'll be trying his technique out later today, but for now I don't see a reason why it wouldn't work. Thus, tentatively marked INVALID.

Sorry for the confusion.

 --downs


-- 

d-bugmail@puremagic.com wrote:
> http://d.puremagic.com/issues/show_bug.cgi?id=1786
> 
> 
> default_357-line@yahoo.de changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>              Status|NEW                         |RESOLVED
>          Resolution|                            |INVALID
> 
> 
> 
> 
> ------- Comment #2 from default_357-line@yahoo.de  2008-01-18 03:16 -------
> Mikola Lysenko, the author of the Original StackThreads
> (http://assertfalse.com/about.shtml), has pointed out a fascinating way of
> making the GC work despite its shortcomings.
> With this, I have to revise my original opinion - looks like making the GC work
> with StackThreads is possible after all :)
> 
> Goes to show that using absolutes like "can't possibly work" is always a bad
> idea. Stupid me.

Looks like it was a good idea in this case.  It goaded someone into finding a way to do what you wanted to do.  :-)

--bb

Bill Baxter wrote:
> 
> Looks like it was a good idea in this case.  It goaded someone into finding a way to do what you wanted to do.  :-)
> 
> --bb

Hehe :)

Also, I still think the GC should be changed to natively accomodate multiple stacks. The current solution works, but is slower by a factor of five or so.

Tango has stack contexts natively integrated with the GC, and it irks me daily that Phobos is behind in this regard.

 --downs

"downs" <default_357-line@yahoo.de> wrote in message news:fn2l4m$1f9$2@digitalmars.com...
> Bill Baxter wrote:
>>
>> Looks like it was a good idea in this case.  It goaded someone into finding a way to do what you wanted to do.  :-)
>>
>> --bb
>
> Hehe :)
>
> Also, I still think the GC should be changed to natively accomodate
> multiple stacks.
> The current solution works, but is slower by a factor of five or so.
>
> Tango has stack contexts natively integrated with the GC, and it irks me daily that Phobos is behind in this regard.
>
> --downs

Then maybe you should use Tango HURR HURR