I wrote simple proof of concept library. The main aim is to
reduce GC usage and improve data locality by replacing dynamic
array for small immutable arrays.


You can find more info here:
   * wiki - https://bitbucket.org/sibnick/inplacearray/wiki/Home
   * source code - https://bitbucket.org/sibnick/inplacearray/src

I include small demo - word counter. This program builds AA array
(word => counter) for given text file in UTF-8. It is possible
switch between builtin array/string and inplace array by changing
one line code. I used:

   -  unpacked List of all page titles -
https://dumps.wikimedia.org/enwiki/20150304/enwiki-20150304-all-titles.gz
as test sample.
   - Ubuntu 14.04 x64
   - dmd 2.67.0
   - dub build -b release
   - ./inplace_array --DRT-gcopt=profile:1 test_big.txt

Here results.

Buildin string:
GC summary: 1355 MB, 12 GC 5659 ms, Pauses 3116 ms < 1436 ms

Inplace array:
GC summary: 715 MB, 22 GC 5281 ms, Pauses 4207 ms < 938 ms

As you can see it is possible noticable reduce memory allocation
(1355Mb -> 715Mb) and max pause time (1436ms -> 938ms).

So it is possible improve situation with GC from new side.

On Saturday, 18 April 2015 at 10:49:01 UTC, Nikolay wrote:
> I wrote simple proof of concept library. The main aim is to
> reduce GC usage and improve data locality by replacing dynamic
> array for small immutable arrays.
>
>
> You can find more info here:
>    * wiki - https://bitbucket.org/sibnick/inplacearray/wiki/Home
>    * source code - https://bitbucket.org/sibnick/inplacearray/src
>
> I include small demo - word counter. This program builds AA array
> (word => counter) for given text file in UTF-8. It is possible
> switch between builtin array/string and inplace array by changing
> one line code. I used:
>
>    -  unpacked List of all page titles -
> https://dumps.wikimedia.org/enwiki/20150304/enwiki-20150304-all-titles.gz
> as test sample.
>    - Ubuntu 14.04 x64
>    - dmd 2.67.0
>    - dub build -b release
>    - ./inplace_array --DRT-gcopt=profile:1 test_big.txt
>
> Here results.
>
> Buildin string:
> GC summary: 1355 MB, 12 GC 5659 ms, Pauses 3116 ms < 1436 ms
>
> Inplace array:
> GC summary: 715 MB, 22 GC 5281 ms, Pauses 4207 ms < 938 ms
>
> As you can see it is possible noticable reduce memory allocation
> (1355Mb -> 715Mb) and max pause time (1436ms -> 938ms).
>
> So it is possible improve situation with GC from new side.

This reminds me of another useful library here:

https://bitbucket.org/infognition/dstuff/src/

See gcarena.d

> This reminds me of another useful library here:
>
> https://bitbucket.org/infognition/dstuff/src/
>
> See gcarena.d

Thanks for link.

The main difference is that I want elimanate pair pointer+data at all. It is more effective to store small array as value type.

On Saturday, 18 April 2015 at 10:49:01 UTC, Nikolay wrote:
> I wrote simple proof of concept library. The main aim is to
> reduce GC usage and improve data locality by replacing dynamic
> array for small immutable arrays.
>
>
> You can find more info here:
>    * wiki - https://bitbucket.org/sibnick/inplacearray/wiki/Home
>    * source code - https://bitbucket.org/sibnick/inplacearray/src
>
> I include small demo - word counter. This program builds AA array
> (word => counter) for given text file in UTF-8. It is possible
> switch between builtin array/string and inplace array by changing
> one line code. I used:
>
>    -  unpacked List of all page titles -
> https://dumps.wikimedia.org/enwiki/20150304/enwiki-20150304-all-titles.gz
> as test sample.
>    - Ubuntu 14.04 x64
>    - dmd 2.67.0
>    - dub build -b release
>    - ./inplace_array --DRT-gcopt=profile:1 test_big.txt
>
> Here results.
>
> Buildin string:
> GC summary: 1355 MB, 12 GC 5659 ms, Pauses 3116 ms < 1436 ms
>
> Inplace array:
> GC summary: 715 MB, 22 GC 5281 ms, Pauses 4207 ms < 938 ms
>
> As you can see it is possible noticable reduce memory allocation
> (1355Mb -> 715Mb) and max pause time (1436ms -> 938ms).
>
> So it is possible improve situation with GC from new side.

I would make the inplace buffer's size a template parameter :)