On Sunday, 9 May 2021 at 13:53:30 UTC, Vladimir Panteleev wrote:
>
> I don't think that would be enough. The garbage collector needs to know of all threads running D code, so that it can scan their stack and registers, so that it can know about objects the pointer to which exists only in that thread's stack or registers.

That's the unfortunate case with D in that case. The otherwise good FF interoperability of D is hampered by this. Then we are back to if GC pointers should be an own type or not. Time for the D3 fork.

On Sunday, 9 May 2021 at 14:10:48 UTC, IGotD- wrote:
> That's the unfortunate case with D in that case. The otherwise good FF interoperability of D is hampered by this. Then we are back to if GC pointers should be an own type or not. Time for the D3 fork.

I wonder what you think about task-bound GC?

https://forum.dlang.org/post/yqdwgbzkmutjzfdhotst@forum.dlang.org

On Sunday, 9 May 2021 at 16:08:07 UTC, Ola Fosheim Grøstad wrote:
>
> I wonder what you think about task-bound GC?
>
> https://forum.dlang.org/post/yqdwgbzkmutjzfdhotst@forum.dlang.org

I see it as a special case where memory management is bounded to a certain primitive/ways of programming. Also a pool for each possible primitive will increase the meta data.

Then you also mention that when the actor is destroyed no scan is needed because the pool is explicitly destroyed as well. This is basically a form of deterministic cleanup that you would expect in C++ when using unique_ptr for example.

Some in that thread mention that D must head towards RC and that's what my opinion is as well. As Walter do not want to make a special fat pointer type, at least making Phobos/druntime using RC internally would be a step in the right direction.

On Sunday, 9 May 2021 at 16:29:12 UTC, IGotD- wrote:
> I see it as a special case where memory management is bounded to a certain primitive/ways of programming. Also a pool for each possible primitive will increase the meta data.

What do you mean by meta data in this context? I think it should resolve at compile time?

Yes, it does imply a programming model if you want GC, but it should allow the programmer to write his own scheduler for flexibility. You would still need RC between tasks...

> Some in that thread mention that D must head towards RC and that's what my opinion is as well. As Walter do not want to make a special fat pointer type, at least making Phobos/druntime using RC internally would be a step in the right direction.

The problem of not having a fat pointer type is when you have arrays of objects. Then you cannot have a RC counter on a negative offset, so you will be forced to embed it in all objects (whether RC-managed or not).

It can be mitigated by requiring that all owning pointers to arrays must be a RC-slice and not a RC-pointer. Then you have a RC-slice of static length 1. (shrug)

On 5/9/21 3:42 AM, Ola Fosheim Grøstad wrote:

> On Sunday, 9 May 2021 at 10:27:04 UTC, Ali Çehreli wrote:
>> whether the library can disable GC and lets
>> the user manage GC collections explicitly (cooperatively?).
>
> Wouldn't this be annoying for the user of the library? (Or maybe you
> only have a handful users?).

Yes, would be annoying; that's why I am asking what others would think about this. (And yes, there are a handful of users.)

> if you have N cores, create N
> worker threads in the daemon and create N pipes?

That's my idea. The trouble is with the thin library layer: It cannot allocate memory while another one is doing collection. This is handled by D runtime for D threads but foreign threads (one that's created by e.g. the C++ program) cannot be known to the D runtime by-default.

> Then set the count of
> the semaphore to N, so when the semaphore hits 0 all pipes are in use,
> and the calling thread will wait in the API stub (wrapper function)
> until a worker thread is available?

That's an interesting idea! But it should be a little different: "I need to collect garbage; I will wait till all others are quiet (and they should know to wait until the collection is over); I collect garbage. Everybody resume." Still, this would require keeping the D GC disabled and my library handles collections at opportune moments that it creates.

> But maybe you want to get rid of the daemon?

I want the daemon because a full-fledged D library that loaded its own libraries as neede caused issues. Perhaps they were all my mistakes but now that it works this way; the daemon stays. :)

Ali

On 5/9/21 6:51 AM, Vladimir Panteleev wrote:

> In my experience, calling D from C/C++ works fine as long as 1) the D
> runtime is allowed to initialize, and 2) all threads which execute D
> code are registered with the D runtime.

I have the following old branch that tries to fix bugs with attaching and detaching:

  https://github.com/dlang/druntime/pull/1989

I also have this worry that if a foreign thread goes without the knowledge of D runtime, the program will crash because the runtime would try to stop a non-existing thread.

> If the threads don't need to share state, you could just as well spawn
> one subprocess per thread, and let it do its own data processing.

That can be achieved by some changes but the issue is not with the backend (the daemon); rather, the foreign threads that call the thin library layer. This layer does allocate (see below), which may trigger collection.

> Another approach would be to listen on a UNIX socket instead of using a
> pipe, which allows using `accept` to open new communication channels
> on-demand.

Aside: I've been under the impression that there couldn't be new pipes opened but I learned that a file descriptor can be passed to another process over unix domain sockets (only Linux is interesting to me here) and the file descriptor and its "passed copy" can be used like a pipe:


https://stackoverflow.com/questions/2358684/can-i-share-a-file-descriptor-to-another-process-on-linux-or-are-they-local-to-t

I haven't experimented with it yet but I think I will use it because the existing channel I wrote uses File objects of PipeProcess, which places large data on shared memory which is re-opened as more space is needed. (And thank you for your work in std.process and more! :) )

> Perhaps it would be simpler to just write the library part in C / C++ /
> -betterC D.

That's a great idea as well.

> std.mmfile has many lines, but the work it has to do is
> actually quite simple.

I know because I've already written the equivalent of mmfile myself by accident. :D When things didn't work about 2 years ago I suspected bugs in mmfile so I wrote my own wrapper. :)

> The same is true about std.socket. This will
> completely avoid your headache with getting the D runtime / GC to play
> well with the host process's threading model.

Makes sense. I've already turned into someone who uses only D's standard library or writes his own. :) The only third party dependency we have is cmake-d.

Ali

On Sunday, 9 May 2021 at 20:36:39 UTC, Ali Çehreli wrote:
> That's an interesting idea! But it should be a little different: "I need to collect garbage; I will wait till all others are quiet (and they should know to wait until the collection is over); I collect garbage. Everybody resume." Still, this would require keeping the D GC disabled and my library handles collections at opportune moments that it creates.

Yes, one semaphore may use an nonoptimal order when it wakes up threads...
Getting this right takes some time.

On 5/9/21 6:58 PM, Ola Fosheim Grostad wrote:

> Yes, one semaphore may use an nonoptimal order when it wakes up threads...
> Getting this right takes some time.

ReadWriteMutex is promising:

  https://dlang.org/library/core/sync/rwmutex/read_write_mutex.html

The writer would be the garbage collecting thread in this case.

Ali

On Sunday, 9 May 2021 at 20:53:28 UTC, Ali Çehreli wrote:
> On 5/9/21 6:51 AM, Vladimir Panteleev wrote:
>
> > In my experience, calling D from C/C++ works fine as long as
> 1) the D
> > runtime is allowed to initialize, and 2) all threads which
> execute D
> > code are registered with the D runtime.
>
> I have the following old branch that tries to fix bugs with attaching and detaching:
>
>   https://github.com/dlang/druntime/pull/1989
>
> I also have this worry that if a foreign thread goes without the knowledge of D runtime, the program will crash because the runtime would try to stop a non-existing thread.
>

That sounds like a bug? 🤔

On Monday, 10 May 2021 at 03:45:23 UTC, Ali Çehreli wrote:
> https://dlang.org/library/core/sync/rwmutex/read_write_mutex.html
>
> The writer would be the garbage collecting thread in this case.

The implementation seems to use a class monitor an syncronized, does that work with non D threads?