http://d.puremagic.com/issues/show_bug.cgi?id=10906

           Summary: [2.064 git-head] Out of memory compiling Phobos on
                    Windows
           Product: D
           Version: unspecified
          Platform: All
        OS/Version: Windows
            Status: NEW
          Severity: regression
          Priority: P2
         Component: Phobos
        AssignedTo: nobody@puremagic.com
        ReportedBy: cbkbbejeap@mailinator.com


--- Comment #0 from Nick Sabalausky <cbkbbejeap@mailinator.com> 2013-08-26 15:36:56 PDT ---
Using the latest github masters of DMD, druntime and phobos, compiling Phobos on my Windows machine with 4GB RAM fails with "Error: out of memory". Despite that, compiling the same versions inside a Linux (Debian 6) VirtualBox VM *inside the same Windows machine* works fine.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=10906


hsteoh@quickfur.ath.cx changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hsteoh@quickfur.ath.cx


--- Comment #1 from hsteoh@quickfur.ath.cx 2013-08-27 15:13:28 PDT ---
Did you figure out what was causing the OOM? I'll bet it's either std.datetime or std.algorithm (likely the latter if you're not running unittests -- std.datetime is very unittest-heavy, but it's std.algorithm that has exponential template expansion in some places *cough* namely cartesianProduct, written by yours truly).

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=10906



--- Comment #2 from Nick Sabalausky <cbkbbejeap@mailinator.com> 2013-08-27 17:26:58 PDT ---
The OOM is occurring during this (rather large) command:

---------------------------------
dmd -lib -ofphobos.lib -Xfphobos.json -O -release -w -d std\stdio.d std\stdiobase.d  std\string.d std\format.d  std\file.d std\range.d std\array.d std\functional.d std\path.d std\outbuffer.d std\utf.d  std\csv.d std\math.d std\complex.d std\numeric.d std\bigint.d  std\metastrings.d std\bitmanip.d std\typecons.d  std\uni.d std\base64.d std\md5.d std\ascii.d  std\demangle.d std\uri.d std\mmfile.d std\getopt.d std\signals.d std\typetuple.d std\traits.d std\encoding.d std\xml.d  std\random.d  std\exception.d  std\compiler.d std\system.d std\concurrency.d std\datetime.d std\uuid.d std\digest\crc.d std\digest\sha.d std\digest\md.d  std\digest\ripemd.d std\digest\digest.d std\algorithm.d std\variant.d  std\syserror.d std\zlib.d  std\stream.d std\socket.d std\socketstream.d  std\container.d std\conv.d  std\zip.d std\cstream.d std\regex.d  std\stdint.d  std\json.d  std\parallelism.d std\mathspecial.d  std\process.d  crc32.d  std\net\isemail.d std\net\curl.d std\c\process.d std\c\stdlib.d std\c\time.d std\c\stdio.d  std\c\math.d std\c\stdarg.d std\c\stddef.d std\c\fenv.d std\c\string.d  std\c\locale.d std\c\wcharh.d  std\windows\registry.d  std\windows\iunknown.d std\windows\syserror.d std\windows\charset.d  std\c\windows\windows.d std\c\windows\com.d  std\c\windows\winsock.d std\c\windows\stat.d std\internal\processinit.d std\internal\uni.d std\internal\uni_tab.d std\internal\unicode_tables.d  std\internal\digest\sha_SSSE3.d std\internal\math\biguintcore.d  std\internal\math\biguintnoasm.d std\internal\math\biguintx86.d  std\internal\math\gammafunction.d std\internal\math\errorfunction.d  std\internal\windows\advapi32.d etc\c\zlib.d etc\c\curl.d etc\c\sqlite3.d  etc\c\zlib\zlib.lib ..\druntime\lib\druntime.lib
---------------------------------

Beyond that, I really don't know. Maybe the posix makefile splits it into smaller chunks than that? Maybe a compiler change resulted in greater memory usage on Windows? Maybe posix just handles low memory better? I'm not sure I'd even know where to start.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=10906



--- Comment #3 from hsteoh@quickfur.ath.cx 2013-08-27 17:34:02 PDT ---
No, in Linux it also runs that huge compile line.

One way to test is perhaps to copy-n-paste that command, omitting likely culprits like std.algorithm and std.datetime (or std.regex, due to ctRegex, but it's less likely) to see if it makes a difference.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=10906



--- Comment #4 from hsteoh@quickfur.ath.cx 2013-08-27 17:34:48 PDT ---
Oh, and add -c to the command line as well, to prevent it from erroring out from undefined symbols.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=10906



--- Comment #5 from Orvid King <blah38621@gmail.com> 2013-08-27 17:36:32 PDT ---
If the compiler attempts to use more than 2GB of virtual memory (including all loaded libraries), then windows will throw an OOM error for exe's which aren't marked as Large Address Aware. (such as dmd) As far as I know, posix systems don't have this limitation and can use the full 4gb of the 32-bit address space without needing to set a flag on the executable. I've managed to trigger this error compiling one of my larger libraries (the arguments actually have to be written to file before they can be passed to dmd, otherwise the command line is too long) it ended up taking roughly 2.5gb of RAM (after I set the Large Address Aware flag) for 2.63.2 to compile my 3.8mb across 1.3k heavily interdependent modules into a static library. Perhaps the Large Address Aware flag should be set by default on dmd?

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=10906



--- Comment #6 from hsteoh@quickfur.ath.cx 2013-08-27 17:38:09 PDT ---
Another thing to try, based on a wild gueses, is to edit std/algorithm.d and add version(none) to cartesianProduct's unittests. One of them, I believe, has a triple cartesian product, which in the current implementation causes an explosion in the number of templates that must be instantiated.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=10906



--- Comment #7 from Nick Sabalausky <cbkbbejeap@mailinator.com> 2013-08-27 17:43:25 PDT ---
Running the original command but with both std.algorthm and std.datetime removed does succeed. Also, running the same command, but with *only* std.algorthm and std.datetime *also* succeeds. So it would seem to be the combination of everything that pushes it over.

However, it still seems a little strange, though: When I run the original command, with everything, the memory usage only goes up to just over 600MB before failing with "out of memory". And I have a couple gigs free, plus virtual memory. Additionally, it works fine on Linux, on the same physical machine, running *inside* a VM (set up with only 768MB RAM system memory). (FWIW, I *am* still getting the OOM on Windows even when the Linux VM isn't running - so it's definitely not a case of the VM using up too much of the available RAM.)

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=10906



--- Comment #8 from Nick Sabalausky <cbkbbejeap@mailinator.com> 2013-08-27 17:47:36 PDT ---
How do I set the Large Address Aware flag? I'm having trouble finding that info through a web search.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=10906



--- Comment #9 from Nick Sabalausky <cbkbbejeap@mailinator.com> 2013-08-27 18:06:59 PDT ---
Actually, if it works with std.algo and atd.datetime done separately (even without the large address aware flag), and if those are notorious for being large, perhaps I should just toss together a pull request adjusting phobos's makefiles to do those separately? Or are there known issues with that?

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------