Hi,

I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d project with ~20 views (*.dt). On my main computer, I have no problems but my laptop has only 4 gig of RAM and dmd is killed before the end of the compilation because it takes more than 4 gig of RAM.

I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but it doesn't change anything. It does work with the "singleFile" option with dub but it's really long to compile... And the "singleFile" option seems to recompile everything even if I only change one ".d" source file.

Are there other solutions?

Thanks,

Guillaume

Absolutely same problem. -lowmem did not get any affect on my VPS (1GB ram). I was able to compile project only in singleFile mode.

On Wednesday, 22 May 2019 at 13:09:37 UTC, Suliman wrote:
> Absolutely same problem. -lowmem did not get any affect on my VPS (1GB ram). I was able to compile project only in singleFile mode.

In those situations you can go with a swapfile.

fallocate -l 4G swapfile
chmod 600 swapfile
mkswap swapfile

then, when you need the file, just

sudo swapon swapfile

and when you're done

sudo swapoff swapfile

Even on an SSD the performance will be terrible, but it should get the job done.

On Wednesday, 22 May 2019 at 13:25:00 UTC, krzaq wrote:
> On Wednesday, 22 May 2019 at 13:09:37 UTC, Suliman wrote:
>> Absolutely same problem. -lowmem did not get any affect on my VPS (1GB ram). I was able to compile project only in singleFile mode.
>
> In those situations you can go with a swapfile.

Does it circumvent process memory limits? Regardless it's not an option if you're dealing with CI and you don't have the neccessary permissions.

$ dub build
Performing "debug" build using /usr/bin/dmd for x86_64.
cachetools 0.1.2: building configuration "library"...
requests 1.0.8: building configuration "std"...
kameloso 1.1.3+commit.66.g4c07959f: building configuration "posix"...
Killed
/usr/bin/dmd failed with exit code 137.

$ fallocate -l 4G /tmp/swapfile
$ chmod 600 /tmp/swapfile
$ mkswap /tmp/swapfile
Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
no label, UUID=7112294b-812a-4765-a3b9-41aa545daa70
$ sudo swapon /tmp/swapfile
swapon: /tmp/swapfile: insecure file owner 3434, 0 (root) suggested.
swapon: /tmp/swapfile: swapon failed: Operation not permitted

On Wednesday, 22 May 2019 at 14:11:07 UTC, Anonymouse wrote:
> On Wednesday, 22 May 2019 at 13:25:00 UTC, krzaq wrote:
>> On Wednesday, 22 May 2019 at 13:09:37 UTC, Suliman wrote:
>>> Absolutely same problem. -lowmem did not get any affect on my VPS (1GB ram). I was able to compile project only in singleFile mode.
>>
>> In those situations you can go with a swapfile.
>
> Does it circumvent process memory limits? Regardless it's not an option if you're dealing with CI and you don't have the neccessary permissions.
>
> $ dub build
> Performing "debug" build using /usr/bin/dmd for x86_64.
> cachetools 0.1.2: building configuration "library"...
> requests 1.0.8: building configuration "std"...
> kameloso 1.1.3+commit.66.g4c07959f: building configuration "posix"...
> Killed
> /usr/bin/dmd failed with exit code 137.
>
> $ fallocate -l 4G /tmp/swapfile
> $ chmod 600 /tmp/swapfile
> $ mkswap /tmp/swapfile
> Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
> no label, UUID=7112294b-812a-4765-a3b9-41aa545daa70
> $ sudo swapon /tmp/swapfile
> swapon: /tmp/swapfile: insecure file owner 3434, 0 (root) suggested.
> swapon: /tmp/swapfile: swapon failed: Operation not permitted

Unless it's a 32-bit process, then it should help. That's what I did when I built geordi on a 512MB DigitalOcean droplet, and it allowed me to have a successful compilation.

As for CI - I have no suggestions outside of "get a beefier one".

On Wednesday, 22 May 2019 at 14:11:07 UTC, Anonymouse wrote:
> On Wednesday, 22 May 2019 at 13:25:00 UTC, krzaq wrote:
>> On Wednesday, 22 May 2019 at 13:09:37 UTC, Suliman wrote:
>>> Absolutely same problem. -lowmem did not get any affect on my VPS (1GB ram). I was able to compile project only in singleFile mode.
>>
>> In those situations you can go with a swapfile.
>
> Does it circumvent process memory limits? Regardless it's not an option if you're dealing with CI and you don't have the neccessary permissions.
>
> $ fallocate -l 4G /tmp/swapfile
> $ chmod 600 /tmp/swapfile
> $ mkswap /tmp/swapfile
> Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
> no label, UUID=7112294b-812a-4765-a3b9-41aa545daa70
> $ sudo swapon /tmp/swapfile
> swapon: /tmp/swapfile: insecure file owner 3434, 0 (root) suggested.
> swapon: /tmp/swapfile: swapon failed: Operation not permitted

I think it will work if you use sudo on all the commands (fallocate, chmod, mkswap), not just swapon.

But it doesn't work for my case since I cannot have a swapfile on my weird setup.

Other ideas?

On Wed, May 22, 2019 at 12:49:56PM +0000, Guillaume via Digitalmars-d wrote:
> Hi,
> 
> I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d project with ~20 views (*.dt). On my main computer, I have no problems but my laptop has only 4 gig of RAM and dmd is killed before the end of the compilation because it takes more than 4 gig of RAM.
> 
> I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but it doesn't change anything. It does work with the "singleFile" option with dub but it's really long to compile... And the "singleFile" option seems to recompile everything even if I only change one ".d" source file.
[...]

Use separate compilation. (I don't know if dub support this.)

Also, in my own vibe.d project, I split up the Diet templates across multiple modules, because they are extremely compile-time intensive. For example, I had one set of pages dealing with account creation, another set of pages with the main functionality, so I split them up into two modules -- since it's unlikely I'll be changing both sets of pages at the same time. That, plus separate compilation, cuts compile time by half.  Better yet, I split up front-facing UI code (mainly Diet templates) from backend logic, so that the Diet templates are not recompiled while I'm changing only the backend logic.


T

-- 
Customer support: the art of getting your clients to pay for your own incompetence.

On Wednesday, 22 May 2019 at 16:23:30 UTC, H. S. Teoh wrote:
> On Wed, May 22, 2019 at 12:49:56PM +0000, Guillaume via Digitalmars-d wrote:
>> Hi,
>> 
>> I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d project with ~20 views (*.dt). On my main computer, I have no problems but my laptop has only 4 gig of RAM and dmd is killed before the end of the compilation because it takes more than 4 gig of RAM.
>> 
>> I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but it doesn't change anything. It does work with the "singleFile" option with dub but it's really long to compile... And the "singleFile" option seems to recompile everything even if I only change one ".d" source file.
> [...]
>
> Use separate compilation. (I don't know if dub support this.)
>

You mean compiling each file to create .o and then linking them? dub supports this with the "singleFile" option but recompiles everything each time. What tool do you use to automate compilation and linking?

Guillaume

On Wed, May 22, 2019 at 05:23:19PM +0000, Guillaume via Digitalmars-d wrote:
> On Wednesday, 22 May 2019 at 16:23:30 UTC, H. S. Teoh wrote:
> > On Wed, May 22, 2019 at 12:49:56PM +0000, Guillaume via Digitalmars-d wrote:
> > > Hi,
> > > 
> > > I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d project with ~20 views (*.dt). On my main computer, I have no problems but my laptop has only 4 gig of RAM and dmd is killed before the end of the compilation because it takes more than 4 gig of RAM.
> > > 
> > > I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but it doesn't change anything. It does work with the "singleFile" option with dub but it's really long to compile... And the "singleFile" option seems to recompile everything even if I only change one ".d" source file.
> > [...]
> > 
> > Use separate compilation. (I don't know if dub support this.)
> > 
> 
> You mean compiling each file to create .o and then linking them? dub supports this with the "singleFile" option but recompiles everything each time. What tool do you use to automate compilation and linking?
[...]

IME, dub is useful as a packaging tool but not so much as a build tool. Because of that I have given up on using dub as a build tool, and only use it for pulling in dependencies.  For builds, I use SCons, though you really could any general build system that isn't crippled. Even Makefiles can do it, despite Make's many issues.

Whichever build system you choose, you don't really have to compile *individual* files; you can compile clusters of them into static libraries (the usual recommendation is one static library per package, but that's only a convention; you can break it up however is most advantageous for you), then link them together into the final executable.

For example, for my vibe.d project I split up the source files into one group for backend logic, then a group for base pages (front page and a few other common "utility" pages), another group for creating new accounts, and another group for the main functionality pages. In theory I could also break up the backend logic into multiple groups, but IME Diet templates take long enough to compile that that's not really the bottleneck, so I left it as a single group for the time being.

Be warned, however, that you'll have to dig into some dirty details of how dub works if you decide to go to go this route -- e.g., you may have to parse dub output in order to get import paths, the list of object files to link into the final executable, etc..  It's all scriptable once you figure it out, of course, but you do have to get your hands dirty once.


T

-- 
Give me some fresh salted fish, please.

On Wednesday, 22 May 2019 at 12:49:56 UTC, Guillaume wrote:
> Hi,
>
> I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d project with ~20 views (*.dt). On my main computer, I have no problems but my laptop has only 4 gig of RAM and dmd is killed before the end of the compilation because it takes more than 4 gig of RAM.
>
> I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but it doesn't change anything. It does work with the "singleFile" option with dub but it's really long to compile... And the "singleFile" option seems to recompile everything even if I only change one ".d" source file.
>
> Are there other solutions?
>
> Thanks,
>
> Guillaume

Do you have the same problem with ldc or gdc?
I have a pc with 4 GB as well and dmd is usually the only one that gets OOM killed when compiling big projects.