I am simply trying to unzip a compressed zip file slightly over 1GB. The de-compressed size is about 4 GB.

The code is very similar to what's explained in the documentation [1] and it works for smaller files.

Anyone has a solution? Memory mapping [2] previously solved some part of my issue but expand is still throwing memory allocation failure.

Selim

[1] https://dlang.org/phobos/std_zip.html
[2] https://forum.dlang.org/thread/mfnleztnwrbgivjvzvdp@forum.dlang.org

Did you try the MmFile workaround?

I am simply trying to unzip a compressed zip file slightly over 1GB. The de-compressed size is about 4 GB.

The code is very similar to what's explained in the documentation [1] and it works for smaller files.

Anyone has a solution? Memory mapping [2] previously solved some part of my issue but expand is still throwing memory allocation failure.

Selim

[1] https://dlang.org/phobos/std_zip.html
[2] https://forum.dlang.org/thread/mfnleztnwrbgivjvzvdp@forum.dlang.org

On Friday, 15 October 2021 at 20:41:36 UTC, Selim Ozel wrote:

It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff.

My work around was to call 7z from D Lang and do the compression over there. That worked like a charm.

It seems that zip.d [1] calls uncompress routine from zlib.d [2]. Would calling zlib uncompress by chunks solve this memory issue? Any ideas?

S

[1] https://github.com/dlang/phobos/blob/master/std/zip.d
[2] https://github.com/dlang/phobos/blob/master/std/zlib.d

Create an issue and we can solve it

It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason  to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff.

On 10/24/21 8:00 AM, Selim Ozel wrote:

Wait, x86 is 32-bit. Max address space is 4GB. So maybe it was just trying to use 4GB and running out of memory?

-Steve

On Monday, 25 October 2021 at 20:50:40 UTC, Steven Schveighoffer wrote:

Good catch, but still, should it use so much memory?

On Monday, 25 October 2021 at 22:38:38 UTC, Imperatorn wrote:

Definitely not. It shouldn't use a lot of memory when unzipping as it should be done in chunks!