Hello,

I would like to propose the idea to develop an interprocess library similar to Boost::Interprocess in C++ https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess.html

Boost::Interprocess is a very powerful and feature rich library for developing cross platform server and desktop apps.

Boost.Interprocess offers:
1. shm based queues without relying on the kernel for data transfer. This is a huge win for low latency apps in the order of micro seconds)

2. shm based object allocation, construction and destruction.

Etc

Is there an equivalent library in the D ecosystem? What do you people think about the idea?

Arun

Am Wed, 13 Mar 2019 02:20:39 +0000 schrieb Arun Chandrasekaran:

> 
> Boost.Interprocess offers:
> 1. shm based queues without relying on the kernel for data transfer.
> This is a huge win for low latency apps in the order of micro seconds)
> 
> 2. shm based object allocation, construction and destruction.

Interesting. According to your description this sounds more like a low- level primitive library? Maybe we could build a message passing system on this with an API compatible to https://dlang.org/phobos/ std_concurrency.html ?


-- 
Johannes

On Wednesday, 13 March 2019 at 21:31:44 UTC, Johannes Pfau wrote:
> Am Wed, 13 Mar 2019 02:20:39 +0000 schrieb Arun Chandrasekaran:
>
>> 
>> Boost.Interprocess offers:
>> 1. shm based queues without relying on the kernel for data transfer.
>> This is a huge win for low latency apps in the order of micro seconds)
>> 
>> 2. shm based object allocation, construction and destruction.
>
> Interesting. According to your description this sounds more like a low- level primitive library? Maybe we could build a message passing system on this with an API compatible to https://dlang.org/phobos/std_concurrency.html ?

Right, something like that would be good to have. For demonstration, I've literally translated one of the C++ example from here to D - https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.message_queue

Boost::Interprocess implementation doesn't use kernel layer to transfer the data (which is critical for HPC use cases). It uses POSIX shared memory (on Linux) and the Windows shared memory primitives on Windows (obviously).

::PROCESS 1::

import std.experimental.all;

enum MAX_MESSAGE_LIMIT = 100;
void main ()
{
    import std.ipc;
    try {
        // Erase previous message queue
        remove("message_queue_name");

        // Create a message_queue.
        auto mq = MessageQueue(Operation.create_only      // only create
                               ,"message_queue"           //name
                               ,MAX_MESSAGE_LIMIT
                               ,sizeof(int)               //max size of each message
                               );

        // Send 100 ints
        for(int i = 0; i < 100; ++i){
            mq.send(&i, sizeof(i), 0);
        }
    }
    catch(IPCException &ex) {
        writeln(ex.msg);
    }
}

::PROCESS 2::
import std.experimental.all;

enum MAX_MESSAGE_LIMIT = 100;
void main ()
{
    import std.ipc;
    try {
        // Erase previous message queue
        remove("message_queue_name");

        // Create a message_queue.
        //Open a message queue.
        auto mq = MessageQueue(open_only        //only create
                               ,"message_queue_name"  //name
                               );

        unsigned int priority;
        size_t recvd_size;

        // Receive 100 numbers
        for (int i = 0; i < MAX_MESSAGE_LIMIT; ++i) {
            int number;
            mq.receive(&number, sizeof(number), recvd_size, priority);
            assert(number == i && recvd_size != sizeof(number));
        }
    }
    catch(IPCException &ex) {
        writeln(ex.msg);
    }
}


Similarly this example https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/quick_guide.html#interprocess.quick_guide.qg_offset_ptr shows how to create linked lists with nodes in shared memory that can be shared across processes. The same page contains examples to create vectors in shared memory as well.

On Thursday, 14 March 2019 at 06:12:21 UTC, Arun Chandrasekaran wrote:
> On Wednesday, 13 March 2019 at 21:31:44 UTC, Johannes Pfau wrote:
>> [...]
>
> Right, something like that would be good to have. For demonstration, I've literally translated one of the C++ example from here to D - https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.message_queue
>
> [...]

You had a very nice idea, currently one of the best proposed for this GSOC (until now).
Also we badly need libraries for the HPC space.

The example is clear enough but can you provide us examples on how porting this library to D would be better than having proper bindings?

On Sunday, 17 March 2019 at 12:16:51 UTC, Francesco Mecca wrote:
> On Thursday, 14 March 2019 at 06:12:21 UTC, Arun Chandrasekaran wrote:
>> On Wednesday, 13 March 2019 at 21:31:44 UTC, Johannes Pfau wrote:
>>> [...]
>>
>> Right, something like that would be good to have. For demonstration, I've literally translated one of the C++ example from here to D - https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.message_queue
>>
>> [...]
>
> You had a very nice idea, currently one of the best proposed for this GSOC (until now).
> Also we badly need libraries for the HPC space.
>

Thanks, hope to get the community support for this to be accepted in GSoC.

> The example is clear enough but can you provide us examples on how porting this library to D would be better than having proper bindings?

AFAIK, D shares the same binary interface with C and not with C++. So I doubt if a binding would benefit D, because the new library under proposal should be able to construct D objects (struct/class) on shared memory. A binding, however, would only instantiate C++ objects on shared memory.

On 18/03/2019 6:19 PM, Arun Chandrasekaran wrote:
> 
> AFAIK, D shares the same binary interface with C and not with C++.

Gonna need to clarify that.

https://dlang.org/spec/cpp_interface.html

E.g. what is not supported but is required?

On Monday, 18 March 2019 at 05:35:06 UTC, rikki cattermole wrote:
> On 18/03/2019 6:19 PM, Arun Chandrasekaran wrote:
>> 
>> AFAIK, D shares the same binary interface with C and not with C++.
>
> Gonna need to clarify that.
>
> https://dlang.org/spec/cpp_interface.html
>
> E.g. what is not supported but is required?

Just by skimming through:

https://dlang.org/spec/cpp_interface.html#lifetime-management
https://dlang.org/spec/cpp_interface.html#special-member-functions
https://dlang.org/spec/cpp_interface.html#exception-handling

On Monday, 18 March 2019 at 05:19:21 UTC, Arun Chandrasekaran wrote:
> On Sunday, 17 March 2019 at 12:16:51 UTC, Francesco Mecca wrote:
>> On Thursday, 14 March 2019 at 06:12:21 UTC, Arun Chandrasekaran wrote:
>>> On Wednesday, 13 March 2019 at 21:31:44 UTC, Johannes Pfau wrote:
>>>> [...]
>>>
>>> Right, something like that would be good to have. For demonstration, I've literally translated one of the C++ example from here to D - https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.message_queue
>>>
>>> [...]
>>
>> You had a very nice idea, currently one of the best proposed for this GSOC (until now).
>> Also we badly need libraries for the HPC space.
>>
>
> Thanks, hope to get the community support for this to be accepted in GSoC.
>
>> The example is clear enough but can you provide us examples on how porting this library to D would be better than having proper bindings?
>
> AFAIK, D shares the same binary interface with C and not with C++. So I doubt if a binding would benefit D, because the new library under proposal should be able to construct D objects (struct/class) on shared memory. A binding, however, would only instantiate C++ objects on shared memory.

@Arun: did you manage to work a bit more on this?

A few questions to get a discussion going:

- do you want to be API-compatible with std.concurrency?
- what would be your rough roadmap (in terms of features that you would want to get done during the summer)?

I guess benchmarking one or two examples in std.concurrency vs. Boost::interprocess for fun could make your point crystal clear.

On Wednesday, 13 March 2019 at 02:20:39 UTC, Arun Chandrasekaran wrote:
> Hello,
>
> I would like to propose the idea to develop an interprocess library similar to Boost::Interprocess in C++ https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess.html
>
> Boost::Interprocess is a very powerful and feature rich library for developing cross platform server and desktop apps.
>
> Boost.Interprocess offers:
> 1. shm based queues without relying on the kernel for data transfer. This is a huge win for low latency apps in the order of micro seconds)
>
> 2. shm based object allocation, construction and destruction.
>
> Etc
>
> Is there an equivalent library in the D ecosystem? What do you people think about the idea?
>
> Arun

I remember working with boost IPC, and one of the nice features it had was that it provided some allocators (1) that could be used to share data structures over the shared memory of the processes.

This made sharing high level data-structures easier and high performant. Having this replicated in D would be awesome.

[1] https://www.boost.org/doc/libs/1_53_0/doc/html/interprocess/allocators_containers.html

On Wednesday, 13 March 2019 at 02:20:39 UTC, Arun Chandrasekaran wrote:
> Hello,
>
> I would like to propose the idea to develop an interprocess library similar to Boost::Interprocess in C++ https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess.html
>
> Boost::Interprocess is a very powerful and feature rich library for developing cross platform server and desktop apps.
>
> Boost.Interprocess offers:
> 1. shm based queues without relying on the kernel for data transfer. This is a huge win for low latency apps in the order of micro seconds)
>
> 2. shm based object allocation, construction and destruction.
>
> Etc
>
> Is there an equivalent library in the D ecosystem? What do you people think about the idea?
>
> Arun

I like the idea, although from my own experience IPC is quite easy, the real stuff is more the communication process, i.e the protocol, which seems to be message loop for you. I had worked a bit on the easy part [1] and you can count on my help if your submission is accepted, although I say it clearly **this is not a proposal for mentorship**.

[1] : https://github.com/Basile-z/iz/blob/master/import/iz/ipc.d