I would post this in d.learn, but I suspect there isn't an easy answer so it would be good to have some serious discussion here.

Is there any way to create a thread that is totally free from the garbage collector?

I.e.
Nothing in that thread will ever be scanned by the GC.
Therefore, when the GC "stops the world", that thread can just keep on going.

Obviously one could create a separate process, but it would be nice to have it encapsulated within a single process for optimum speed of communication and not having to mess around with pipes.

P.S. Yes I realise how careful one would have to be when using this.

If you make a thread using the operating system functions directly, D would never know (I'm pretty sure) and thus it won't be on the gc nor the list the gc uses to pause all threads.

On Monday, 8 July 2013 at 14:04:41 UTC, John Colvin wrote:
> I would post this in d.learn, but I suspect there isn't an easy answer so it would be good to have some serious discussion here.
>
> Is there any way to create a thread that is totally free from the garbage collector?
>
> I.e.
> Nothing in that thread will ever be scanned by the GC.
> Therefore, when the GC "stops the world", that thread can just keep on going.
>
> Obviously one could create a separate process, but it would be nice to have it encapsulated within a single process for optimum speed of communication and not having to mess around with pipes.
>
> P.S. Yes I realise how careful one would have to be when using this.


We really need to implement a better GC. That's the very hard but better solution.

On Monday, 8 July 2013 at 14:11:16 UTC, w0rp wrote:
> On Monday, 8 July 2013 at 14:04:41 UTC, John Colvin wrote:
>> I would post this in d.learn, but I suspect there isn't an easy answer so it would be good to have some serious discussion here.
>>
>> Is there any way to create a thread that is totally free from the garbage collector?
>>
>> I.e.
>> Nothing in that thread will ever be scanned by the GC.
>> Therefore, when the GC "stops the world", that thread can just keep on going.
>>
>> Obviously one could create a separate process, but it would be nice to have it encapsulated within a single process for optimum speed of communication and not having to mess around with pipes.
>>
>> P.S. Yes I realise how careful one would have to be when using this.
>
>
> We really need to implement a better GC. That's the very hard but better solution.

We need to have a <b>choice</b> of GCs. There is no single design that is right for every use case. Sometimes (admittedly rather rarely) conservative stop-the-world is actually a good thing!

On Monday, 8 July 2013 at 14:16:46 UTC, John Colvin wrote:
> On Monday, 8 July 2013 at 14:11:16 UTC, w0rp wrote:
>>
>> We really need to implement a better GC. That's the very hard but better solution.
>
> We need to have a <b>choice</b> of GCs. There is no single design that is right for every use case. Sometimes (admittedly rather rarely) conservative stop-the-world is actually a good thing!

Yes, this is true. Java offers a few options for tweaking the GC, so we could do the same.

On Monday, 8 July 2013 at 14:04:41 UTC, John Colvin wrote:
> Is there any way to create a thread that is totally free from the garbage collector?

Sure, just few pull request here, some pull requests there... ;)

On Monday, 8 July 2013 at 14:11:15 UTC, Adam D. Ruppe wrote:
> If you make a thread using the operating system functions directly, D would never know (I'm pretty sure) and thus it won't be on the gc nor the list the gc uses to pause all threads.

Huh. Well that was easy. Presumably that means I have to resort to doing all my communication with OS-level functions (e.g. pthread_****).

I wonder if one could somehow register a pre-existing thread with std.concurrency, being careful not to introduce any reference that lets in the garbage collector.

On Monday, 8 July 2013 at 14:53:25 UTC, John Colvin wrote:
> I wonder if one could somehow register a pre-existing thread with std.concurrency, being careful not to introduce any reference that lets in the garbage collector.

There is a thread_attachThis() in core.thread that "Registers the calling thread for use with the D Runtime. " and returns a Thread object.

However, I think that registers it with the GC too so it defeats the purpose of skipping Thread in the first place.

Looking at the source of std.concurrency, it looks like it just has a thread local mailbox for each one, that is created when you call thisTid() the first time.

idk if that's actually all that is done, or if TLS needs any special work (I've never actually used pthreads directly), but maybe you could try calling that thisTid() function in the thread you make yourself and then send it a message and see what happens.

On Jul 8, 2013, at 8:08 AM, Adam D. Ruppe <destructionator@gmail.com> wrote:

> On Monday, 8 July 2013 at 14:53:25 UTC, John Colvin wrote:
>> I wonder if one could somehow register a pre-existing thread with std.concurrency, being careful not to introduce any reference that lets in the garbage collector.
> 
> There is a thread_attachThis() in core.thread that "Registers the calling thread for use with the D Runtime. " and returns a Thread object.
> 
> However, I think that registers it with the GC too so it defeats the purpose of skipping Thread in the first place.

Right.  The point of that routine is to make it so an external thread can safely reference garbage collected memory.


> Looking at the source of std.concurrency, it looks like it just has a thread local mailbox for each one, that is created when you call thisTid() the first time.
> 
> idk if that's actually all that is done, or if TLS needs any special work (I've never actually used pthreads directly), but maybe you could try calling that thisTid() function in the thread you make yourself and then send it a message and see what happens.

You'd really need to call thread_attachThis() first, since std.concurrency uses garbage collected memory for message passing.

On Jul 8, 2013, at 7:04 AM, John Colvin <john.loughran.colvin@gmail.com> wrote:

> I would post this in d.learn, but I suspect there isn't an easy answer so it would be good to have some serious discussion here.
> 
> Is there any way to create a thread that is totally free from the garbage collector?
> 
> I.e.
> Nothing in that thread will ever be scanned by the GC.
> Therefore, when the GC "stops the world", that thread can just keep on going.
> 
> Obviously one could create a separate process, but it would be nice to have it encapsulated within a single process for optimum speed of communication and not having to mess around with pipes.
> 
> P.S. Yes I realise how careful one would have to be when using this.


There's a pull request for this that's been discussed on the Druntime list and while I'm willing to accept it so long as sufficient safeguards are in place, I still really don't understand the need for this feature.  Once you eliminate the GC support code there really isn't much left in core.thread.  What advantage is there of being able to create Thread objects that aren't managed by the GC (which I'll call juggernaut threads)?