This is NOT flamebait!

I like D.  I want the answer to the question posed in the subject of this post to be a hearty and resounding "yes!"  However, I have some concerns that such won't be the case.

D is a GC'd language through and through, and I'm not interested in trying to extirpate that quality.  I imagine that if I wrote a program that didn't tickle the GC in any way, shape, or form (i.e. 100% static memory), then it would be Hard-RT compliant.  However, it probably wouldn't be terribly useful.

The specific scenario I'm interested in is multi-threaded programming with one Hard-RT thread, communicating with the non-RT threads via a lock-free ring-buffer.  The Hard-RT thread would be very well behaved and not do one iota of memory allocation.  Conversely, the other threads, especially the GUI thread, would.

The D spec says that all threads must pause when the GC runs, regardless of which thread it runs in.  This seems to invalidate my design's RT-ness.  The obvious solution of making the program multiprocess (barf) isn't too useful, because that would make adjusting the RT thread's memory allocation a rather convoluted process.  With the current design, I can adjust the RT thread's memory structures in one of the non-RT threads, and protect the data with a mutex that is try_locked().  A multi-process solution would be markedly uglier, and I can't think of any method that would be 100% correct too boot; tickling the GC seems inevitable.

Is there any way around this?  Is it even possible to write a GC that doesn't stop up all threads?

<ninjadroid@gazuga.net> wrote in message news:chqq71$k0o$1@digitaldaemon.com...
> This is NOT flamebait!
>
> I like D.  I want the answer to the question posed in the subject of this
post
> to be a hearty and resounding "yes!"  However, I have some concerns that
such
> won't be the case.
>
> D is a GC'd language through and through, and I'm not interested in trying
to
> extirpate that quality.  I imagine that if I wrote a program that didn't
tickle
> the GC in any way, shape, or form (i.e. 100% static memory), then it would
be
> Hard-RT compliant.  However, it probably wouldn't be terribly useful.
>
> The specific scenario I'm interested in is multi-threaded programming with
one
> Hard-RT thread, communicating with the non-RT threads via a lock-free ring-buffer.  The Hard-RT thread would be very well behaved and not do one
iota
> of memory allocation.  Conversely, the other threads, especially the GUI
thread,
> would.
>
> The D spec says that all threads must pause when the GC runs, regardless
of
> which thread it runs in.  This seems to invalidate my design's RT-ness.
The
> obvious solution of making the program multiprocess (barf) isn't too
useful,
> because that would make adjusting the RT thread's memory allocation a
rather
> convoluted process.  With the current design, I can adjust the RT thread's memory structures in one of the non-RT threads, and protect the data with
a
> mutex that is try_locked().  A multi-process solution would be markedly
uglier,
> and I can't think of any method that would be 100% correct too boot;
tickling
> the GC seems inevitable.
>
> Is there any way around this?  Is it even possible to write a GC that
doesn't
> stop up all threads?
>

You can still use C's malloc() and free(), and create threads that are
separate from the GC using C functions, but you can't easily use the GC with
them.

On Thu, 9 Sep 2004 23:49:53 +0000 (UTC), ninjadroid@gazuga.net wrote:

>This is NOT flamebait!
>
>I like D.  I want the answer to the question posed in the subject of this post to be a hearty and resounding "yes!"  However, I have some concerns that such won't be the case.
>
>D is a GC'd language through and through, and I'm not interested in trying to extirpate that quality.  I imagine that if I wrote a program that didn't tickle the GC in any way, shape, or form (i.e. 100% static memory), then it would be Hard-RT compliant.  However, it probably wouldn't be terribly useful.
>
>The specific scenario I'm interested in is multi-threaded programming with one Hard-RT thread, communicating with the non-RT threads via a lock-free ring-buffer.  The Hard-RT thread would be very well behaved and not do one iota of memory allocation.  Conversely, the other threads, especially the GUI thread, would.
>
>The D spec says that all threads must pause when the GC runs, regardless of which thread it runs in.  This seems to invalidate my design's RT-ness.  The obvious solution of making the program multiprocess (barf) isn't too useful, because that would make adjusting the RT thread's memory allocation a rather convoluted process.  With the current design, I can adjust the RT thread's memory structures in one of the non-RT threads, and protect the data with a mutex that is try_locked().  A multi-process solution would be markedly uglier, and I can't think of any method that would be 100% correct too boot; tickling the GC seems inevitable.
>
>Is there any way around this?  Is it even possible to write a GC that doesn't stop up all threads?
>

I think i found in the manual that the GC cycle cannot start if no allocation is done anywhere in the code. So maybe you can allocate all memory you need before the RT condition starts. So with a careful design you can make sure the gc doesn't get called during RT critical time slices. if you need to allocate, you can disable gc before and enable after doing so. Or you can allocate with malloc, the gc will ignore that but then you have to clean up yourself.

At a first glance it should not be very difficult to build hard RT conditions into your D-code, but how do you want to obtain hard RT conditions from your OS? Check this out: http://www.flounder.com/time.htm - if that is true you will have to boot your machine with your program, in order to obtain hard RT conditions.

ninjadroid@gazuga.net wrote:

> This is NOT flamebait!
> 
> I like D.  I want the answer to the question posed in the subject of this
> post
> to be a hearty and resounding "yes!"  However, I have some concerns that
> such won't be the case.
> 
> D is a GC'd language through and through, and I'm not interested in trying
> to
> extirpate that quality.  I imagine that if I wrote a program that didn't
> tickle the GC in any way, shape, or form (i.e. 100% static memory), then
> it would be
> Hard-RT compliant.  However, it probably wouldn't be terribly useful.
> 
> The specific scenario I'm interested in is multi-threaded programming with
> one Hard-RT thread, communicating with the non-RT threads via a lock-free
> ring-buffer.  The Hard-RT thread would be very well behaved and not do one
> iota
> of memory allocation.  Conversely, the other threads, especially the GUI
> thread, would.
> 
> The D spec says that all threads must pause when the GC runs, regardless
> of
> which thread it runs in.  This seems to invalidate my design's RT-ness.
> The obvious solution of making the program multiprocess (barf) isn't too
> useful, because that would make adjusting the RT thread's memory
> allocation a rather
> convoluted process.  With the current design, I can adjust the RT thread's
> memory structures in one of the non-RT threads, and protect the data with
> a
> mutex that is try_locked().  A multi-process solution would be markedly
> uglier, and I can't think of any method that would be 100% correct too
> boot; tickling the GC seems inevitable.
> 
> Is there any way around this?  Is it even possible to write a GC that doesn't stop up all threads?

sure - that wouldn't be hard at all. I would modify two files: std/thread.d
and internal/gc/gcx.d
To thread.d I'd add a flag indicating if the thread is not to be stopped
during GC (and ignore such threads during pauseAll) and to gcx.d in the
function fullCollect I'd weed out those threads that are flagged. It would
be up to the programmer to make sure the non-GC thread doesn't contain any
sole references to GC-managed data. It would probably be a while before
anything like this made it into D proper - if at all because it is trusting
the programmer quite a lot to mark a thread as invisible to the GC.

Am I understanding the situation - or are there GC-managed objects that have the only live reference from the RT thread?

-Ben

<ninjadroid@gazuga.net> wrote in message news:chqq71$k0o$1@digitaldaemon.com...
> The D spec says that all threads must pause when the GC runs, regardless
of
> which thread it runs in.  This seems to invalidate my design's RT-ness.
The
> obvious solution of making the program multiprocess (barf) isn't too
useful,
> because that would make adjusting the RT thread's memory allocation a
rather
> convoluted process.  With the current design, I can adjust the RT thread's memory structures in one of the non-RT threads, and protect the data with
a
> mutex that is try_locked().  A multi-process solution would be markedly
uglier,
> and I can't think of any method that would be 100% correct too boot;
tickling
> the GC seems inevitable.
>
> Is there any way around this?  Is it even possible to write a GC that
doesn't
> stop up all threads?

The gc only pauses all threads that it knows about. There's nothing stopping a D program from bypassing std.thread and creating a thread directly from the operating system APIs. That thread wouldn't be paused during a gc collection cycle, however, it would be incumbent upon the programmer to make *sure* that the unpaused thread does not hold any references to objects that *are not also held by any paused threads*.

D, being a systems programming language, lets you cheat and break the rules when necessary.

WOB wrote:

> At a first glance it should not be very difficult to build hard RT conditions into your D-code, but how do you want to obtain hard RT conditions from your OS? Check this out: http://www.flounder.com/time.htm - if that is true you will have to boot your machine with your program, in order to obtain hard RT conditions.

Depends on whether "your OS" really refers to Windows as you silently
assume. There is a number of operating systems out there that support hard
RT. One example is RTLinux, a modification of the original Linux kernel.
There even is a free version of that to check out:
 http://www.fsmlabs.com/products/openrtlinux/

On Fri, 10 Sep 2004 08:46:15 +0200, Norbert Nemec <Norbert@Nemec-online.de> wrote:

>Depends on whether "your OS" really refers to Windows as you silently assume. There is a number of operating systems out there that support hard RT.

Well I didn't silently assume Windows but yes I did silently assume any kind of general-purpose OS since there was also a reference to GUI threads in the original post, but no reference to RT constraints of the OS. But you are right, I should have written "... with your own program, or install a real time OS..."

In article <chr729$uvr$1@digitaldaemon.com>, Walter says...
>
>
>The gc only pauses all threads that it knows about. There's nothing stopping a D program from bypassing std.thread and creating a thread directly from the operating system APIs. That thread wouldn't be paused during a gc collection cycle, however, it would be incumbent upon the programmer to make *sure* that the unpaused thread does not hold any references to objects that *are not also held by any paused threads*.

OK, so to rephrase for those of us (i.e. me) that don't follow this kind of
stuff too well:

If I create a thread via a direct call to the OS, the GC won't know about it. If the GC doesn't know about the thread, it won't stop it during a collection run.  However, I need to then make sure that all the allocated memory the "renegade" thread refers to is also referenced by a thread that the GC *is* aware of; otherwise, the GC will think that memory is actually unused, and possibly make my life miserable as a result.

Somewhat OT question: in a GC thread, will calls to malloc() register with the
GC?  That is, if you allocate memory with malloc(), you don't have to worry
about the GC gobbling it up?

>D, being a systems programming language, lets you cheat and break the rules when necessary.

That is *music* to my ears!

In article <chsal1$1guv$1@digitaldaemon.com>, ninjadroid@gazuga.net says...
>
>Somewhat OT question: in a GC thread, will calls to malloc() register with the
>GC?  That is, if you allocate memory with malloc(), you don't have to worry
>about the GC gobbling it up?

No.  Though it's worth noting that most memory allocators have critical sections.  It may be worth writing your own allocator or seeing if you can integrate an aftermarket version if you are concerned about this bottleneck.

>That is *music* to my ears!

Where is BeOS when you need it? ;)


Sean

In article <chsal1$1guv$1@digitaldaemon.com>, ninjadroid@gazuga.net wrote:

> In article <chr729$uvr$1@digitaldaemon.com>, Walter says...
> >
> >
> >The gc only pauses all threads that it knows about. There's nothing stopping a D program from bypassing std.thread and creating a thread directly from the operating system APIs. That thread wouldn't be paused during a gc collection cycle, however, it would be incumbent upon the programmer to make *sure* that the unpaused thread does not hold any references to objects that *are not also held by any paused threads*.
> 
> OK, so to rephrase for those of us (i.e. me) that don't follow this kind of
> stuff too well:
> 
> If I create a thread via a direct call to the OS, the GC won't know about it. If the GC doesn't know about the thread, it won't stop it during a collection run.  However, I need to then make sure that all the allocated memory the "renegade" thread refers to is also referenced by a thread that the GC *is* aware of; otherwise, the GC will think that memory is actually unused, and possibly make my life miserable as a result.


"it would be incumbent upon the programmer to make sure* that the unpaused thread does _NOT_ hold any references to objects that are not also held by any paused threads*."


> 
> Somewhat OT question: in a GC thread, will calls to malloc() register with the
> GC?  That is, if you allocate memory with malloc(), you don't have to worry
> about the GC gobbling it up?
> 
> >D, being a systems programming language, lets you cheat and break the rules when necessary.
> 
> That is *music* to my ears!