Hi,
i want to duplicate/clone a thread with the stack and tls, like the standard function fork() in c. Is in the phobos library a method that do the same thing?

Gianni Pisetta

On 12/10/2013 04:04 AM, Gianni Pisetta wrote:
> Hi,
> i want to duplicate/clone a thread with the stack and tls, like the
> standard function fork() in c. Is in the phobos library a method that do
> the same thing?
>
> Gianni Pisetta

Apparently, there isn't. However, you can use fork() directly from the core.sys.posix.unistd module.

Ali

On Tuesday, 10 December 2013 at 13:39:17 UTC, Ali Çehreli wrote:
> On 12/10/2013 04:04 AM, Gianni Pisetta wrote:
>> Hi,
>> i want to duplicate/clone a thread with the stack and tls, like the
>> standard function fork() in c. Is in the phobos library a method that do
>> the same thing?
>>
>> Gianni Pisetta
>
> Apparently, there isn't. However, you can use fork() directly from the core.sys.posix.unistd module.
>
> Ali

I had the same idea, but assumed (maybe wrongly) that it messes up the garbage collection because it creates a copy of the allocated objects in tls that the gc is not aware of. I searched the core.thread reference to see if it has a similar method but found none. Then i posted this question. So is it safe to use fork()? I now realized that i don't know how the gc works.

Gianni Pisetta

On 12/10/2013 06:56 AM, Gianni Pisetta wrote:
> On Tuesday, 10 December 2013 at 13:39:17 UTC, Ali Çehreli wrote:
>> On 12/10/2013 04:04 AM, Gianni Pisetta wrote:
>>> Hi,
>>> i want to duplicate/clone a thread with the stack and tls, like the
>>> standard function fork() in c. Is in the phobos library a method that do
>>> the same thing?
>>>
>>> Gianni Pisetta
>>
>> Apparently, there isn't. However, you can use fork() directly from the
>> core.sys.posix.unistd module.
>>
>> Ali
>
> I had the same idea, but assumed (maybe wrongly) that it messes up the
> garbage collection because it creates a copy of the allocated objects in
> tls that the gc is not aware of. I searched the core.thread reference to
> see if it has a similar method but found none. Then i posted this
> question. So is it safe to use fork()? I now realized that i don't know
> how the gc works.
>
> Gianni Pisetta

I don't know such details but I presume fork() is safe. Grepping in Phobos modules, process.d uses it.

Ali

On Tuesday, 10 December 2013 at 17:14:47 UTC, Ali Çehreli wrote:
> On 12/10/2013 06:56 AM, Gianni Pisetta wrote:
>> On Tuesday, 10 December 2013 at 13:39:17 UTC, Ali Çehreli wrote:
>>> On 12/10/2013 04:04 AM, Gianni Pisetta wrote:
>>>> Hi,
>>>> i want to duplicate/clone a thread with the stack and tls, like the
>>>> standard function fork() in c. Is in the phobos library a method that do
>>>> the same thing?
>>>>
>>>> Gianni Pisetta
>>>
>>> Apparently, there isn't. However, you can use fork() directly from the
>>> core.sys.posix.unistd module.
>>>
>>> Ali
>>
>> I had the same idea, but assumed (maybe wrongly) that it messes up the
>> garbage collection because it creates a copy of the allocated objects in
>> tls that the gc is not aware of. I searched the core.thread reference to
>> see if it has a similar method but found none. Then i posted this
>> question. So is it safe to use fork()? I now realized that i don't know
>> how the gc works.
>>
>> Gianni Pisetta
>
> I don't know such details but I presume fork() is safe. Grepping in Phobos modules, process.d uses it.
>
> Ali

Ok, thanks for the reply. I will give it a try.

Gianni Pisetta

Am Tue, 10 Dec 2013 20:36:11 +0100
schrieb "Gianni Pisetta" <pisetta.gianni@alice.it>:

> On Tuesday, 10 December 2013 at 17:14:47 UTC, Ali Çehreli wrote:
> > On 12/10/2013 06:56 AM, Gianni Pisetta wrote:
> >> On Tuesday, 10 December 2013 at 13:39:17 UTC, Ali Çehreli wrote:
> >>> On 12/10/2013 04:04 AM, Gianni Pisetta wrote:
> >>>> Hi,
> >>>> i want to duplicate/clone a thread with the stack and tls,
> >>>> like the
> >>>> standard function fork() in c. Is in the phobos library a
> >>>> method that do
> >>>> the same thing?
> >>>>
> >>>> Gianni Pisetta
> >>>
> >>> Apparently, there isn't. However, you can use fork() directly
> >>> from the
> >>> core.sys.posix.unistd module.
> >>>
> >>> Ali
> >>
> >> I had the same idea, but assumed (maybe wrongly) that it
> >> messes up the
> >> garbage collection because it creates a copy of the allocated
> >> objects in
> >> tls that the gc is not aware of. I searched the core.thread
> >> reference to
> >> see if it has a similar method but found none. Then i posted
> >> this
> >> question. So is it safe to use fork()? I now realized that i
> >> don't know
> >> how the gc works.
> >>
> >> Gianni Pisetta
> >
> > I don't know such details but I presume fork() is safe. Grepping in Phobos modules, process.d uses it.
> >
> > Ali
> 
> Ok, thanks for the reply. I will give it a try.
> 
> Gianni Pisetta

D is cross-platform, so to have a wrapper for fork() would
mean to get it working on Windows, too.
std.process, can must use fork() of course, since it is the
way to spawn a child process on Posix systems. fork() doesn't
clone a thread, but a whole process. So there is no issue
with the GC, as it will be cloned as well. Until Posix threads
were introduced, it was common to create "threads" using
fork(), so it probably works ok for you. Maybe we can find a
better solution though if you tell us some more.

For example this exists in D:

static this()
{
  // Initialize TLS variables
}

That is a module constructor that is run once for each new
Thread. You could prepare your TLS variables there as a copy
of global data or other function calls.
If you really need an automated copy of all TLS variables for
a new thread the situation looks dim I guess :p

-- 
Marco

> D is cross-platform, so to have a wrapper for fork() would
> mean to get it working on Windows, too.
> std.process, can must use fork() of course, since it is the
> way to spawn a child process on Posix systems. fork() doesn't
> clone a thread, but a whole process. So there is no issue
> with the GC, as it will be cloned as well. Until Posix threads
> were introduced, it was common to create "threads" using
> fork(), so it probably works ok for you. Maybe we can find a
> better solution though if you tell us some more.
>
> For example this exists in D:
>
> static this()
> {
>   // Initialize TLS variables
> }
>
> That is a module constructor that is run once for each new
> Thread. You could prepare your TLS variables there as a copy
> of global data or other function calls.
> If you really need an automated copy of all TLS variables for
> a new thread the situation looks dim I guess :p

No, i think fork or something similar is the way i prefer to go.
I'm working on a simple top-down parser, so i save all the data references in the context of the function and at the end of it i create the syntax tree object with all his childrens. So i have the already allocated objects in the tls and the info on the realtionships of these objects plus the parser state in the call stack. I want to duplicate the parser because i want all the possible syntax trees as a result and when the tokenizer will find in the input two or more possible tokens, it must fork and return for each thread one token from the possible ones. At the end if one or more threads made to the end, each thread copy his syntax tree on a shared array and the main thread then can work on the results.

Another way to solve this is to move all the info from the call stack to a stack managed by me, have a standard procedure that fills the stack. When i must duplicate the thread i can create a new one, make a deep copy of the objects in the stack and the stack itself, then call the standard procedure again. But i think this is a bottom-up parser and a major change in the application design.

For now i will stick with fork, if in the future a similar function is implemented in core.thread, i will use it.

Am Wed, 11 Dec 2013 09:10:09 +0100
schrieb "Gianni Pisetta" <pisetta.gianni@alice.it>:

> No, i think fork or something similar is the way i prefer to go. I'm working on a simple top-down parser, so i save all the data references in the context of the function and at the end of it i create the syntax tree object with all his childrens. So i have the already allocated objects in the tls and the info on the realtionships of these objects plus the parser state in the call stack. I want to duplicate the parser because i want all the possible syntax trees as a result and when the tokenizer will find in the input two or more possible tokens, it must fork and return for each thread one token from the possible ones. At the end if one or more threads made to the end, each thread copy his syntax tree on a shared array and the main thread then can work on the results.
> 
> Another way to solve this is to move all the info from the call stack to a stack managed by me, have a standard procedure that fills the stack. When i must duplicate the thread i can create a new one, make a deep copy of the objects in the stack and the stack itself, then call the standard procedure again. But i think this is a bottom-up parser and a major change in the application design.
>
> For now i will stick with fork, if in the future a similar function is implemented in core.thread, i will use it.

Alright, fork is optimized for this kind of stuff and it should work fine. Personally I would likely have tried a manged stack instead of cloning the whole application. How do you write back the results from the child processes? Do you use pipes or shared memory?

-- 
Marco

On Wednesday, 11 December 2013 at 08:54:18 UTC, Marco Leise wrote:
> Am Wed, 11 Dec 2013 09:10:09 +0100
> schrieb "Gianni Pisetta" <pisetta.gianni@alice.it>:
>
>> No, i think fork or something similar is the way i prefer to go.
>> I'm working on a simple top-down parser, so i save all the data references in the context of the function and at the end of it i create the syntax tree object with all his childrens. So i have the already allocated objects in the tls and the info on the realtionships of these objects plus the parser state in the call stack. I want to duplicate the parser because i want all the possible syntax trees as a result and when the tokenizer will find in the input two or more possible tokens, it must fork and return for each thread one token from the possible ones. At the end if one or more threads made to the end, each thread copy his syntax tree on a shared array and the main thread then can work on the results.
>> 
>> Another way to solve this is to move all the info from the call stack to a stack managed by me, have a standard procedure that fills the stack. When i must duplicate the thread i can create a new one, make a deep copy of the objects in the stack and the stack itself, then call the standard procedure again. But i think this is a bottom-up parser and a major change in the application design.
>>
>> For now i will stick with fork, if in the future a similar function is implemented in core.thread, i will use it.
>
> Alright, fork is optimized for this kind of stuff and it
> should work fine. Personally I would likely have tried a
> manged stack instead of cloning the whole application. How do
> you write back the results from the child processes? Do you
> use pipes or shared memory?

I also like the managed stack idea, but i think the main problem with that is that it is harder to understand what's going on when you have to debug.
I'm not at the point yet of passing the results to the main thread, but i think at the end i will make the entire tree immutable(or maybe already create it with immutable) and add the reference to a shared array or use a message passing pattern like the one in std.concurrency.

Gianni Pisetta

Am Wed, 11 Dec 2013 11:04:29 +0100
schrieb "Gianni Pisetta" <pisetta.gianni@alice.it>:

> On Wednesday, 11 December 2013 at 08:54:18 UTC, Marco Leise wrote:
> > Am Wed, 11 Dec 2013 09:10:09 +0100
> > schrieb "Gianni Pisetta" <pisetta.gianni@alice.it>:
> >
> >> No, i think fork or something similar is the way i prefer to
> >> go.
> >> I'm working on a simple top-down parser, so i save all the
> >> data references in the context of the function and at the end
> >> of it i create the syntax tree object with all his childrens.
> >> So i have the already allocated objects in the tls and the
> >> info on the realtionships of these objects plus the parser
> >> state in the call stack. I want to duplicate the parser
> >> because i want all the possible syntax trees as a result and
> >> when the tokenizer will find in the input two or more possible
> >> tokens, it must fork and return for each thread one token from
> >> the possible ones. At the end if one or more threads made to
> >> the end, each thread copy his syntax tree on a shared array
> >> and the main thread then can work on the results.
> >> 
> >> Another way to solve this is to move all the info from the call stack to a stack managed by me, have a standard procedure that fills the stack. When i must duplicate the thread i can create a new one, make a deep copy of the objects in the stack and the stack itself, then call the standard procedure again. But i think this is a bottom-up parser and a major change in the application design.
> >>
> >> For now i will stick with fork, if in the future a similar function is implemented in core.thread, i will use it.
> >
> > Alright, fork is optimized for this kind of stuff and it should work fine. Personally I would likely have tried a manged stack instead of cloning the whole application. How do you write back the results from the child processes? Do you use pipes or shared memory?
> 
> I also like the managed stack idea, but i think the main problem
> with that is that it is harder to understand what's going on when
> you have to debug.
> I'm not at the point yet of passing the results to the main
> thread, but i think at the end i will make the entire tree
> immutable(or maybe already create it with immutable) and add the
> reference to a shared array or use a message passing pattern like
> the one in std.concurrency.
> 
> Gianni Pisetta

Then wait a second! fork() creates a new _process_, not a thread. The two options I gave are pretty much the only ones available for communication between local processes. Maybe add a TCP socket connection to that...

Shared arrays and std.concurrency work only for _threads_ inside the same process.

-- 
Marco