Hi,

I am migrating a project to Windows.
DMD fails with

Fatal error: out of memory

Splitting the project to dozen subpackages allows to workaround the issue. In other hand, dub test doesn't test subpackages.

It would be nice to have 64 bit DMD.

Best Regards,
Ilya Yaroshenko

On Friday, 29 June 2018 at 14:39:29 UTC, 9il wrote:
> Hi,
>
> I am migrating a project to Windows.
> DMD fails with
>
> Fatal error: out of memory
>
> Splitting the project to dozen subpackages allows to workaround the issue. In other hand, dub test doesn't test subpackages.
>
> It would be nice to have 64 bit DMD.
>
> Best Regards,
> Ilya Yaroshenko

You can grab one at AppVeyor https://ci.appveyor.com/project/greenify/dmd/build/1.0.4829/artifacts (it gets build with every PR/commit at DMD).
I hope we manage to ship it with the official releases soon.

On Friday, 29 June 2018 at 16:11:56 UTC, Seb wrote:
> I hope we manage to ship it with the official releases soon.

I can't wait.

Honestly, a compiler should never run out of memory and it's a major issue that DMD does it.

Regardless of that it has fast compilation, it should never run out of memory.

I assume it's something that will never be fixed in DMD however, because of its memory model.

On Friday, June 29, 2018 20:23:05 bauss via Digitalmars-d wrote:
> On Friday, 29 June 2018 at 16:11:56 UTC, Seb wrote:
> > I hope we manage to ship it with the official releases soon.
>
> I can't wait.
>
> Honestly, a compiler should never run out of memory and it's a major issue that DMD does it.
>
> Regardless of that it has fast compilation, it should never run out of memory.
>
> I assume it's something that will never be fixed in DMD however, because of its memory model.

It could be fixed, even if it's just adding a compiler switch to indicate that you want it to use the GC to collect memory, because you're going to run out of memory if you don't. As I understand it, the compiler already can have that turned on internally. It's just that it hurt compilation times enough to do so that it's been left off - which makes sense for many, many cases, but is a definite problem when your project is large enough or doing enough CTFE that compiling it causes it to run out of memory. What the best way to handle ti is, I don't know, since we do want the compilation speed in the typical case, but we also don't want to run out of memory. A compiler switch would be an easy out but less than ideal.

Regardless of all of that, I expect that newCTFE will have a fairly large impact on memory consumption once it's finally ready, because the current CTFE is an absolute memory hog (e.g. as I understand it, it allocates a new object every time a variable is mutated, so doing something like incrementing an integer in a loop in CTFE means that the compiler is doing a _lot_ of allocating). So, depending on why folks run out of memory when compiling, newCTFE alone may fix the problem for most folks.

Still, we really should have the 64-bit compiler binary available on Windows. For most developers, there's really no reason to use a 32-bit binary, and if it solves the running out of memory problem (even if it's only by being able to throw more memory at the compiler), then that will be a big help even if it doesn't fix the underlying problem that the compiler isn't versatile enough to be able to free memory when it needs to without paying the full cost of memory management when it doesn't need to.

- Jonathan M Davis

On Friday, 29 June 2018 at 20:57:24 UTC, Jonathan M Davis wrote:
> [...]
>
> It could be fixed, even if it's just adding a compiler switch to indicate that you want it to use the GC to collect memory, because you're going to run out of memory if you don't. As I understand it, the compiler already can have that turned on internally. It's just that it hurt compilation times enough to do so that it's been left off - which makes sense for many, many cases, but is a definite problem when your project is large enough or doing enough CTFE that compiling it causes it to run out of memory. What the best way to handle ti is, I don't know, since we do want the compilation speed in the typical case, but we also don't want to run out of memory. A compiler switch would be an easy out but less than ideal.
>
> [...]

I managed to get around this issue by using LDC to compile a test project, but the memory usage was insane. It used more than 4GB of memory. I would rather have slower compile times, personally. I bet letting the GC do its job and free memory is less time consuming than letting the compiler eventually run out of memory, halt, and leave me to scramble to free up system memory so that I can try again.

On Fri, Jun 29, 2018 at 11:41:41PM +0000, SonicFreak94 via Digitalmars-d wrote:
> On Friday, 29 June 2018 at 20:57:24 UTC, Jonathan M Davis wrote:
> > [...]
> > It could be fixed, even if it's just adding a compiler switch to
> > indicate that you want it to use the GC to collect memory, because
> > you're going to run out of memory if you don't. As I understand it,
> > the compiler already can have that turned on internally. It's just
> > that it hurt compilation times enough to do so that it's been left
> > off - which makes sense for many, many cases, but is a definite
> > problem when your project is large enough or doing enough CTFE that
> > compiling it causes it to run out of memory.

This has been a showstopper for me on low-memory machines. If I hadn't been using a high-memory system when I first tried out D, I might have just walked away.

This *needs* to be an option.

Though the last time I tried compiling DMD with the GC enabled, it segfaulted on me, so there may be bugs that need to be fixed in order for this to work.


[...]
> I managed to get around this issue by using LDC to compile a test project, but the memory usage was insane. It used more than 4GB of memory. I would rather have slower compile times, personally. I bet letting the GC do its job and free memory is less time consuming than letting the compiler eventually run out of memory, halt, and leave me to scramble to free up system memory so that I can try again.

Absolutely agree.  As I said, this issue has been a showstopper for me on low-memory machines; it basically makes D unusable. (And one of the environments I was trying to use dmd in is a remote colocated server for which adding additional memory is not a practical option.)


T

-- 
It only takes one twig to burn down a forest.

On 30/06/2018 8:57 AM, Jonathan M Davis wrote:
> It could be fixed, even if it's just adding a compiler switch to indicate
> that you want it to use the GC to collect memory

Currently leaks memory like crazy.

On Friday, 29 June 2018 at 23:57:51 UTC, H. S. Teoh wrote:
> This has been a showstopper for me on low-memory machines. If I hadn't been using a high-memory system when I first tried out D, I might have just walked away.

Confirmed. On a Raspberry Pi D compilers are completely useless.

For example, I can compile huge C projects like the PHP interpreter without a problem (it's horribly slow, though of course 😌), but I run out of memory with anything a bit bigger written in D.

Well, this applies to LDC, at least (I've got no plan if the LLVM backend frees memory). Haven't tried GDC yet (distro packages are as outdated as GDC's releases, and I'm too lazy to compile that thing myself). DMD is not available for ARM.


In my humble opinion, there should be a (compile-time?) GC switch for the frontend.

Sorry if this post has any wrong assumptions. Feel free to correct me :)


Kind regards,
Elias

On Saturday, 30 June 2018 at 22:38:13 UTC, 0xEAB wrote:
> On Friday, 29 June 2018 at 23:57:51 UTC, H. S. Teoh wrote:
>> This has been a showstopper for me on low-memory machines. If I hadn't been using a high-memory system when I first tried out D, I might have just walked away.
>
> Confirmed. On a Raspberry Pi D compilers are completely useless.
>
> For example, I can compile huge C projects like the PHP interpreter without a problem (it's horribly slow, though of course 😌), but I run out of memory with anything a bit bigger written in D.
>
> Well, this applies to LDC, at least (I've got no plan if the LLVM backend frees memory).

LDC uses the same GC-less D frontend as DMD, that's likely the issue, not the llvm backend.