On 28/06/18 01:46, Steven Schveighoffer wrote:
> This has been a thorn in many sides for a long time. I remember Weka.io's Liran talking about how they required an INSANE amount of time/memory to build their system in dconf 2015 maybe? But things have gotten a bit better since then. I think at some point there will be a reckoning where it has to be fixed. Fast compile-time is cool, but inifinte compile time (i.e. it never finishes because the OOM killer destroys your process) is not acceptable.

No, we just use bigger build servers:
$ cat /proc/cpuinfo
...
processor       : 127
vendor_id       : GenuineIntel
cpu family      : 6
model           : 63
model name      : Intel(R) Xeon(R) CPU E7-8880 v3 @ 2.30GHz
stepping        : 4
microcode       : 0x11
cpu MHz         : 2300.072
cache size      : 46080 KB
physical id     : 3
siblings        : 32
core id         : 15
cpu cores       : 16
apicid          : 223
initial apicid  : 223
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq monitor est ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm ida xsaveopt fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm
bogomips        : 4662.97
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

ubuntu@ip-172-31-14-32:~$ cat /proc/meminfo
MemTotal:       2014743200 kB


My laptop has 32GB of ram. If I do single threaded compilation and allocate enough swap, I can *barely* squeeze through a debug build compilation of the node.

Shachar

On Wednesday, 27 June 2018 at 16:00:37 UTC, Mario Silva wrote:
> Less than an year ago our project was taking around 17 seconds to compile - no libs requiring compilation - and maybe around 50 seconds for full compilation, and it now takes around 50 seconds for an incremental compilations and around 1.5 minutes for a full one.

If memory consumption has got worse in recent releases, it's possible it was my fault. I made a pull* to fix it, so it might be worth trying the 2.081.0 beta as it should have the fix merged.

* https://github.com/dlang/dmd/pull/8281

On 2018-06-27 18:00, Mario Silva wrote:
> Hello,
> 
> Our code base has been growing steadily and it's currently at a point where my 16GB machine just freezes when we're compiling our code. This happens because DMD just consumes all my memory for a while.
> 
> Also, it's taking a long time to compile it. Less than an year ago our project was taking around 17 seconds to compile - no libs requiring compilation - and maybe around 50 seconds for full compilation, and it now takes around 50 seconds for an incremental compilations and around 1.5 minutes for a full one.
> 
> For you guys to have an idea of the size of our project, we have 21151 lines of code and then 50933 in our libs. This is just our code without counting dependencies fetched by dub like vibe.d for example.
> 
> Are there any plans to work on compiler performance in terms of memory consumption and compilation time?
> 
> Any tips on how to code in a way that minimizes both compilation times and memory consumption when compiling?

Unless this already has been stated, the issue is usually not the size of the project, it's rather which compile time features which are used. For example the DWT project has over 200k lines of code and compile (to a library) in a couple of seconds. Try to reduce templates and CTFE code, if possible and prefer CTFE over templates.

-- 
/Jacob Carlborg

On 06/27/2018 12:00 PM, Mario Silva wrote:
> Hello,
> 
> Our code base has been growing steadily and it's currently at a point where my 16GB machine just freezes when we're compiling our code. This happens because DMD just consumes all my memory for a while.
> 
> Also, it's taking a long time to compile it. Less than an year ago our project was taking around 17 seconds to compile - no libs requiring compilation - and maybe around 50 seconds for full compilation, and it now takes around 50 seconds for an incremental compilations and around 1.5 minutes for a full one.
> 

Memory consumption is a known issue with the D frontend implementation. As others have said, CTFE never freeing memory is considered to be a big part of the issue here. The CTFE rewrite someone is doing should help with that when it's done.

Long compilation times are almost always the fault of Dub. Dub is pretty well-known for taking DMD's near-instant compile times and bloating them out to the ridiculous (by D standards) lengths you're experiencing. I'd say try just writing a basic buildscript that runs DMD directly. Plus, that way you can customize it to your own project's needs: Which files/packages get compiled together, which get compiled separately.

On Wed, Jul 04, 2018 at 04:17:22PM -0400, Nick Sabalausky (Abscissa) via Digitalmars-d wrote:
[...]
> Long compilation times are almost always the fault of Dub. Dub is pretty well-known for taking DMD's near-instant compile times and bloating them out to the ridiculous (by D standards) lengths you're experiencing. I'd say try just writing a basic buildscript that runs DMD directly. Plus, that way you can customize it to your own project's needs: Which files/packages get compiled together, which get compiled separately.

Yeah, I'm sorry to say that I have been quite disappointed with dub (as a build system), and currently have taken to avoiding it as much as possible. As a packaging system it's not bad, but it simply doesn't have the options I need to use it as a build tool.

But to be fair, it's not always dub's fault.  Heavy template / CTFE usage is also known to bloat compilation times, sometimes by a lot. Earlier this year I discovered that the very act of importing std.format or std.regex causes compilation time to almost double.  There were some issues related to compiling with -unittest that triggered this behaviour, though we found that the underlying issue was actually more tricky than it might first appear.  I don't remember if this issue has been completely resolved yet.


T

-- 
Живёшь только однажды.