Thread overview
std.zip and a large archive
Jul 19, 2014
AntonSotov
Jul 19, 2014
Marc Schütz
Jul 19, 2014
AntonSotov
Jul 19, 2014
bearophile
Re: std.zip and a large archive (spell fix)
July 19, 2014
I process archive:
///////////////
import std.stdio, std.zip, std.file;

int main()
{
  auto zip = new ZipArchive(read("c:/test.zip"));
  foreach (item; zip.directory) {
    writeln("processing ", item.name, " ...");
    // processing item...
  }
  return 0;
}
/////////////////
it works well for normal archives.
but how to process zip archive ~1GB ?
it takes a long of RAM.
July 19, 2014
On Saturday, 19 July 2014 at 07:55:10 UTC, AntonSotov wrote:
> I process archive:
> ///////////////
> import std.stdio, std.zip, std.file;
>
> int main()
> {
>   auto zip = new ZipArchive(read("c:/test.zip"));
>   foreach (item; zip.directory) {
>     writeln("processing ", item.name, " ...");
>     // processing item...
>   }
>   return 0;
> }
> /////////////////
> it works well for normal archives.
> but how to process zip archive ~1GB ?
> it takes a long of RAM.

Hmm... it's unfortunate that ZipArchive doesn't take a file descriptor. As a workaround, you can use memory mapping:

import std.stdio, std.zip, std.file, std.mmfile;

int main()
{
  auto mmfile = new MmFile(File("c:/test.zip", "rb"));
  auto zip = new ZipArchive(mmfile[]);
  foreach (item; zip.directory) {
    writeln("processing ", item.name, " ...");
    // processing item...
  }
  return 0;
}
July 19, 2014
On Saturday, 19 July 2014 at 10:46:31 UTC, Marc Schütz wrote:

> Hmm... it's unfortunate that ZipArchive doesn't take a file descriptor. As a workaround, you can use memory mapping:

auto mmfile = new MmFile("c:/test.zip");
Thank you. it works!

July 19, 2014
Marc Schütz:

> import std.stdio, std.zip, std.file, std.mmfile;
>
> int main()
> {
>   auto mmfile = new MmFile(File("c:/test.zip", "rb"));
>   auto zip = new ZipArchive(mmfile[]);
>   foreach (item; zip.directory) {
>     writeln("processing ", item.name, " ...");
>     // processing item...
>   }
>   return 0;
> }

This could be added to the ZipArchive online docs.

Bye,
bearophile
July 19, 2014
It seems that ti only works for zip files with less than 65000 entries, zip64 that allow read more than that do seem to be implemented.
July 19, 2014
It seems that this only works for zip files with less than 65000
entries, zip64 that allow read more than that do not seem to be
implemented.