I, due to a need, will start implementation of distributed memory system.

Idea is that:

Let's say you have allocated 1 GiB space in memory. This memory is blocked into 4 KiB.

After some reservation, and free operations, now only the blocks 0, 12, and 13 are free to be allocated.

Problem is that those memory blocks are not consecutive.

With the implementation of distributed memory, those blocks will be appended into a struct or object (however it is implemented), and that struct or object will allow access to that 12 KiB of space (3 x 4 KiB blocks) like it is allocated in memory consecutively.

Is there anything like this in Phobos, or shall I start my own implementation?

On Monday, 18 January 2016 at 05:59:15 UTC, tcak wrote:
> I, due to a need, will start implementation of distributed memory system.
>
> Idea is that:
>
> Let's say you have allocated 1 GiB space in memory. This memory is blocked into 4 KiB.
>
> After some reservation, and free operations, now only the blocks 0, 12, and 13 are free to be allocated.
>
> Problem is that those memory blocks are not consecutive.
>
> With the implementation of distributed memory, those blocks will be appended into a struct or object (however it is implemented), and that struct or object will allow access to that 12 KiB of space (3 x 4 KiB blocks) like it is allocated in memory consecutively.
>
> Is there anything like this in Phobos, or shall I start my own implementation?

Check https://dlang.org/phobos/std_experimental_allocator.html

On Monday, 18 January 2016 at 08:12:03 UTC, Nemanja Boric wrote:
> On Monday, 18 January 2016 at 05:59:15 UTC, tcak wrote:
>> [...]
>
> Check https://dlang.org/phobos/std_experimental_allocator.html

Which part of this module provide the functionality of using non-consecutive memory(distributed) blocks like they are consecutive?

On Monday, 18 January 2016 at 05:59:15 UTC, tcak wrote:
> I, due to a need, will start implementation of distributed memory system.
>
> Idea is that:
>
> Let's say you have allocated 1 GiB space in memory. This memory is blocked into 4 KiB.
>
> After some reservation, and free operations, now only the blocks 0, 12, and 13 are free to be allocated.
>
> Problem is that those memory blocks are not consecutive.
>
> With the implementation of distributed memory, those blocks will be appended into a struct or object (however it is implemented), and that struct or object will allow access to that 12 KiB of space (3 x 4 KiB blocks) like it is allocated in memory consecutively.
>
> Is there anything like this in Phobos, or shall I start my own implementation?

Just a note about terminology: I'd rather call it "fragmented memory" or something alike. The term "distributed memory" usually refers to something really different [1].

Your idea seems interesting, but IMHO a compacting GC should be the preferred solution for heap fragmentation.

[1] https://en.wikipedia.org/wiki/Distributed_memory

On Monday, 18 January 2016 at 09:56:17 UTC, Adrian Matoga wrote:
> On Monday, 18 January 2016 at 05:59:15 UTC, tcak wrote:
>> I, due to a need, will start implementation of distributed memory system.
>>
>> Idea is that:
>>
>> Let's say you have allocated 1 GiB space in memory. This memory is blocked into 4 KiB.
>>
>> After some reservation, and free operations, now only the blocks 0, 12, and 13 are free to be allocated.
>>
>> Problem is that those memory blocks are not consecutive.
>>
>> With the implementation of distributed memory, those blocks will be appended into a struct or object (however it is implemented), and that struct or object will allow access to that 12 KiB of space (3 x 4 KiB blocks) like it is allocated in memory consecutively.
>>
>> Is there anything like this in Phobos, or shall I start my own implementation?
>
> Just a note about terminology: I'd rather call it "fragmented memory" or something alike. The term "distributed memory" usually refers to something really different [1].
>
> Your idea seems interesting, but IMHO a compacting GC should be the preferred solution for heap fragmentation.
>
> [1] https://en.wikipedia.org/wiki/Distributed_memory

I was trying to remember the Windows program that is about disk clusters "De...". Couldn't remember. It was Defragmentation. Yes, Fragmented Memory is much better. It is definitely not that distributed memory thing.

In my use case, a server program reserves a part of shared memory to client application, then when the client program is done with that memory, it releases them. So, making it dependent on GC might not be good. But I am sure, many use cases can be found that fits to GC.

On Monday, 18 January 2016 at 09:28:59 UTC, tcak wrote:
> On Monday, 18 January 2016 at 08:12:03 UTC, Nemanja Boric wrote:
>> On Monday, 18 January 2016 at 05:59:15 UTC, tcak wrote:
>>> [...]
>>
>> Check https://dlang.org/phobos/std_experimental_allocator.html
>
> Which part of this module provide the functionality of using non-consecutive memory(distributed) blocks like they are consecutive?

IIRC, none of them, sorry, but if you're going to implement it, my guess it that
it should be compatible with `std.experimental.allocator`.

On Monday, 18 January 2016 at 11:46:36 UTC, Nemanja Boric wrote:
> On Monday, 18 January 2016 at 09:28:59 UTC, tcak wrote:
>> On Monday, 18 January 2016 at 08:12:03 UTC, Nemanja Boric wrote:
>>> Check https://dlang.org/phobos/std_experimental_allocator.html
>>
>> Which part of this module provide the functionality of using non-consecutive memory(distributed) blocks like they are consecutive?
>
> IIRC, none of them, sorry, but if you're going to implement it, my guess it that
> it should be compatible with `std.experimental.allocator`.

Allocators work with void[] arrays which require memory to be contiguous. This idea of fragmented memory simply isn't compatible with allocators. Such a library could provide a range interface but it would have to be wrapped in a container of some sort. It could potentially be a candidate for std.container though.

On Monday, 18 January 2016 at 09:56:17 UTC, Adrian Matoga wrote:
> ...
> Your idea seems interesting, but IMHO a compacting GC should be the preferred solution for heap fragmentation.

Implementing a compacting GC in D would be exceedingly difficult, if not impossible, because of raw pointers, unions, non-GC memory, linked C/C++ code, and possibly other reasons. Other languages, such as Java, have a much simpler model that can be dealt with. There's simply too many caveats in D and any such benefits would be minimal at best.

On Monday, 18 January 2016 at 05:59:15 UTC, tcak wrote:
> Is there anything like this in Phobos, or shall I start my own implementation?

It isn't really clear to me what you are trying to do. IIRC the C++ deque is usually implemented as an array of pointers to fixed sized memory blocks.

Is that what you are trying to achieve?

On Tuesday, 19 January 2016 at 10:09:01 UTC, Ola Fosheim Grøstad wrote:
> On Monday, 18 January 2016 at 05:59:15 UTC, tcak wrote:
>> Is there anything like this in Phobos, or shall I start my own implementation?
>
> It isn't really clear to me what you are trying to do. IIRC the C++ deque is usually implemented as an array of pointers to fixed sized memory blocks.
>
> Is that what you are trying to achieve?

First of all, I have started implementation, and a part of implementation is completed.

For your question, I can explain it as follows:

Example I have 3 memory blocks.

Memory block 1: Ptr = 0x1000, Len = 100
Memory block 2: Ptr = 0x2000, Len = 5
Memory block 3: Ptr = 0x3000, Len = 150

Into the class FragmentedMemory, you append those three memory blocks. Then it provides you an interface like those three memory blocks are consecutive.

Let's say: fragmem[ 125 ] = 5;

Because the index 125 is in memory block 3, the value is written to memoryBlock3[ 20 ];
(125 = 100 + 5 + 20 )

Reading is in the same way. I think you can think about use cases on your side.

Currently, set, get, append operations are completed.

I will implement bulk memory copy and assign operations as well.