std.zip expand: memory allocation failed (page 2)

ubyte[] expand(ArchiveMember de); Decompress the contents of a member. Fills in properties extractVersion, flags, compressionMethod, time, crc32, compressedSize, expandedSize, expandedData[], name[], extra[].

On Tuesday, 26 October 2021 at 20:33:17 UTC, Steven Schveighoffer wrote:

On 10/26/21 1:38 PM, Imperatorn wrote:

That's the current implementation.

No, that's the API. You cannot fix the implementation with that API and not end up allocating an array to hold the entire unzipped contents. You can't even decompress to a file, and then mmap those contents -- the address space isn't there.

I don't know about *nix, but my Windows machine can easily extract a file bigger than my RAM.

zlib does not require decompressing in entirety. This is just the way that std.zip decided to expose the API. e.g. iopipe uses an expandable buffer for decompression and compression, which does not have to contain the entire file.

-Steve

Yes, that's how it's written. That's the problem.

October 27, 2021

Re: std.zip expand: memory allocation failed

Posted by bauss
in reply to Steven Schveighoffer

Permalink

bauss

Posted in reply to Steven Schveighoffer

Permalink

On Tuesday, 26 October 2021 at 13:43:36 UTC, Steven Schveighoffer wrote:

On 10/26/21 2:32 AM, bauss wrote:

On Monday, 25 October 2021 at 22:38:38 UTC, Imperatorn wrote:

On Monday, 25 October 2021 at 20:50:40 UTC, Steven Schveighoffer wrote:

On 10/24/21 8:00 AM, Selim Ozel wrote:

It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff.

Wait, x86 is 32-bit. Max address space is 4GB. So maybe it was just trying to use 4GB and running out of memory?

-Steve

Good catch, but still, should it use so much memory?

Definitely not. It shouldn't use a lot of memory when unzipping as it should be done in chunks!

You guys aren't getting it:

ubyte[] expand(ArchiveMember de);
Decompress the contents of a member.
Fills in properties extractVersion, flags, compressionMethod, time, crc32, compressedSize, expandedSize, expandedData[], name[], extra[].

Where is it supposed to store that ubyte[]?

-Steve

It's not supposed, but a new implementation can utilize something like a stream.

Forums