Might be of use to someone, but I was looking for ways to speed up dmd's albeit already fast compilation times.

Just by dropping in jemalloc in place of glibc's malloc via LD_PRELOAD on my linux machine I saw a 10-15% drop in compilation times across the board. Not sure if this is common knowledge or not.

(Sorry if this takes away from your coffee time : ) )

On Thursday, 22 January 2015 at 19:37:11 UTC, weaselcat wrote:
> Might be of use to someone, but I was looking for ways to speed up dmd's albeit already fast compilation times.
>
> Just by dropping in jemalloc in place of glibc's malloc via LD_PRELOAD on my linux machine I saw a 10-15% drop in compilation times across the board. Not sure if this is common knowledge or not.
>
> (Sorry if this takes away from your coffee time : ) )

I thought dmd used a bump-the-pointer approach, since it almost never frees memory?

On 1/22/15 11:37 AM, weaselcat wrote:
> Might be of use to someone, but I was looking for ways to speed up dmd's
> albeit already fast compilation times.
>
> Just by dropping in jemalloc in place of glibc's malloc via LD_PRELOAD
> on my linux machine I saw a 10-15% drop in compilation times across the
> board. Not sure if this is common knowledge or not.
>
> (Sorry if this takes away from your coffee time : ) )

I think we should statically link jemalloc with dmd. Any takers? -- Andrei

On Thu, 22 Jan 2015 19:39:24 +0000, Vladimir Panteleev wrote:

> On Thursday, 22 January 2015 at 19:37:11 UTC, weaselcat wrote:
>> Might be of use to someone, but I was looking for ways to speed up dmd's albeit already fast compilation times.
>>
>> Just by dropping in jemalloc in place of glibc's malloc via LD_PRELOAD on my linux machine I saw a 10-15% drop in compilation times across the board. Not sure if this is common knowledge or not.
>>
>> (Sorry if this takes away from your coffee time : ) )
> 
> I thought dmd used a bump-the-pointer approach, since it almost never frees memory?

it not frees, but it still does alot of `new`. seems that jmalloc is better at it.

On Thu, Jan 22, 2015 at 07:59:23PM +0000, ketmar via Digitalmars-d wrote:
> On Thu, 22 Jan 2015 19:39:24 +0000, Vladimir Panteleev wrote:
> 
> > On Thursday, 22 January 2015 at 19:37:11 UTC, weaselcat wrote:
> >> Might be of use to someone, but I was looking for ways to speed up dmd's albeit already fast compilation times.
> >>
> >> Just by dropping in jemalloc in place of glibc's malloc via LD_PRELOAD on my linux machine I saw a 10-15% drop in compilation times across the board. Not sure if this is common knowledge or not.
> >>
> >> (Sorry if this takes away from your coffee time : ) )
> > 
> > I thought dmd used a bump-the-pointer approach, since it almost never frees memory?
> 
> it not frees, but it still does alot of `new`. seems that jmalloc is better at it.

Huh?! I thought dmd just uses a bump-the-pointer allocation?  How can that possibly be slower that whatever jmalloc does?


T

-- 
Тише едешь, дальше будешь.

On Thursday, 22 January 2015 at 19:59:23 UTC, ketmar wrote:
> On Thu, 22 Jan 2015 19:39:24 +0000, Vladimir Panteleev wrote:
>
>> On Thursday, 22 January 2015 at 19:37:11 UTC, weaselcat wrote:
>>> Might be of use to someone, but I was looking for ways to speed up
>>> dmd's albeit already fast compilation times.
>>>
>>> Just by dropping in jemalloc in place of glibc's malloc via LD_PRELOAD
>>> on my linux machine I saw a 10-15% drop in compilation times across the
>>> board. Not sure if this is common knowledge or not.
>>>
>>> (Sorry if this takes away from your coffee time : ) )
>> 
>> I thought dmd used a bump-the-pointer approach, since it almost never
>> frees memory?
>
> it not frees, but it still does alot of `new`. seems that jmalloc is
> better at it.

https://github.com/D-Programming-Language/dmd/blob/master/src/root/rmem.c

Looks like it only does that for C++ `new`.

On Thursday, 22 January 2015 at 19:37:11 UTC, weaselcat wrote:
> Might be of use to someone, but I was looking for ways to speed up dmd's albeit already fast compilation times.
>
> Just by dropping in jemalloc in place of glibc's malloc via LD_PRELOAD on my linux machine I saw a 10-15% drop in compilation times across the board. Not sure if this is common knowledge or not.
>
> (Sorry if this takes away from your coffee time : ) )

I had some time to bench it on a project not mine

I used vibe.d BTW

avgtime -d -q -r10 dub build --compiler=dmd --force

------------------------
Total time (ms): 73167.1
Repetitions    : 10
Sample mode    : 7290 (3 ocurrences)
Median time    : 7300.78
Avg time       : 7316.71
Std dev.       : 32.5878
Minimum        : 7281.8
Maximum        : 7385.81
95% conf.int.  : [7252.84, 7380.59]  e = 63.871
99% conf.int.  : [7232.77, 7400.65]  e = 83.9407
EstimatedAvg95%: [7296.52, 7336.91]  e = 20.1978
EstimatedAvg99%: [7290.17, 7343.26]  e = 26.5444

LD_PRELOAD=/usr/lib/libjemalloc.so avgtime -d -q -r10 dub build --compiler=dmd --force

------------------------
Total time (ms): 64026.4
Repetitions    : 10
Sample mode    : 6390 (3 ocurrences)
Median time    : 6399.68
Avg time       : 6402.64
Std dev.       : 18.3341
Minimum        : 6365.8
Maximum        : 6431.14
95% conf.int.  : [6366.7, 6438.57]  e = 35.9342
99% conf.int.  : [6355.41, 6449.86]  e = 47.2256
EstimatedAvg95%: [6391.27, 6414]  e = 11.3634
EstimatedAvg99%: [6387.7, 6417.57]  e = 14.934



p.s., avgtime from dub is a nice tool.

On Thursday, 22 January 2015 at 19:37:11 UTC, weaselcat wrote:
> Might be of use to someone, but I was looking for ways to speed up dmd's albeit already fast compilation times.
>
> Just by dropping in jemalloc in place of glibc's malloc via LD_PRELOAD on my linux machine I saw a 10-15% drop in compilation times across the board. Not sure if this is common knowledge or not.
>
Might be interesting to try some other implementations, too?  TCMalloc[0] and nedmalloc[1] at least have permissive licenses.  (Hoard[2] and Lockless[3] also seem interesting, but are GPL or worse.)

-Wyatt

[0] Part of gperf-tools https://code.google.com/p/gperftools/
[1] http://www.nedprod.com/programs/portable/nedmalloc/index.html
[2] http://www.hoard.org/
[3] http://locklessinc.com/downloads/

On 1/23/15 8:59 AM, Wyatt wrote:
> On Thursday, 22 January 2015 at 19:37:11 UTC, weaselcat wrote:
>> Might be of use to someone, but I was looking for ways to speed up
>> dmd's albeit already fast compilation times.
>>
>> Just by dropping in jemalloc in place of glibc's malloc via LD_PRELOAD
>> on my linux machine I saw a 10-15% drop in compilation times across
>> the board. Not sure if this is common knowledge or not.
>>
> Might be interesting to try some other implementations, too? TCMalloc[0]
> and nedmalloc[1] at least have permissive licenses. (Hoard[2] and
> Lockless[3] also seem interesting, but are GPL or worse.)
>
> -Wyatt
>
> [0] Part of gperf-tools https://code.google.com/p/gperftools/
> [1] http://www.nedprod.com/programs/portable/nedmalloc/index.html
> [2] http://www.hoard.org/
> [3] http://locklessinc.com/downloads/

Definitely measurements should help. From the many measurements we've run at Facebook, jemalloc would be overall a good bet. Also I work with Jason Evans so if there are issues we want improved I might ask him :o). -- Andrei

On Friday, 23 January 2015 at 16:59:41 UTC, Wyatt wrote:
> On Thursday, 22 January 2015 at 19:37:11 UTC, weaselcat wrote:
>> Might be of use to someone, but I was looking for ways to speed up dmd's albeit already fast compilation times.
>>
>> Just by dropping in jemalloc in place of glibc's malloc via LD_PRELOAD on my linux machine I saw a 10-15% drop in compilation times across the board. Not sure if this is common knowledge or not.
>>
> Might be interesting to try some other implementations, too?  TCMalloc[0] and nedmalloc[1] at least have permissive licenses.
>  (Hoard[2] and Lockless[3] also seem interesting, but are GPL or worse.)
>
> -Wyatt
>
> [0] Part of gperf-tools https://code.google.com/p/gperftools/
> [1] http://www.nedprod.com/programs/portable/nedmalloc/index.html
> [2] http://www.hoard.org/
> [3] http://locklessinc.com/downloads/

Hi,
I actually did try tcmalloc and I can't remember if it performed better than malloc or not, but it performed overall worse than jemalloc. I'm assuming because it's(as the name impllies), optimized for heavily threaded scenarios which DMD is not.
I didn't try the others because they're not readily available in my package manager.