Greetings!

I have this program that zips a file and everything works perfectly, if the
files are small enough.  But, I am having to zip files that are getting more
and more extreme in size and I am running out of memory.  Here is the output
when executing a small program to zip a folder:

 0:36:59.76>zipafile
Zipping file no 0 huge\0.pdf
 ** PLEASE WAIT FOR FILE #0 TO BE ZIPPED **
Zipping file no 1 huge\1.exe
 ** PLEASE WAIT FOR FILE #1 TO BE ZIPPED **
Zipping file no 2 huge\2.exe
 ** PLEASE WAIT FOR FILE #2 TO BE ZIPPED **
Zipping file no 3 huge\updates\p.zip
 ** PLEASE WAIT FOR FILE #3 TO BE ZIPPED **
Zipping file no 4 huge\updates\q.html
 ** PLEASE WAIT FOR FILE #4 TO BE ZIPPED **
Zipping file no 5 huge\updates\z.exe
 ** PLEASE WAIT FOR FILE #5 TO BE ZIPPED **
finished all files...
 ** PLEASE WAIT FOR ZIP FILE TO BE CREATED **
Error: Out of memory

 0:37:56.26>

This is the program in question...

import std.stdio;
import std.file;
import std.date;
import std.zip;
import std.zlib;
import jic.libs.MyFile;

int main(char[][] args)
{

  char[] folder = r"c:\tmp\huge";
  ZipAFolder(folder);
  return 0;
}

void ZipAFolder(char[] zipFolder)
{
  char[] fdname = std.path.getDirName(zipFolder);
  char[][] allfiles = std.file.listdir(zipFolder,"*");
  char[] zipFile = zipFolder ~ ".zip";
  int FilesCnt = 0;

  std.zip.ZipArchive zr;
  zr = new std.zip.ZipArchive();
  int one = 0;
  foreach(char[] f; allfiles)
  {
    char[] f0 = std.string.replace(f,fdname ~ "\\","");  // Filename
    writefln("Zipping file no " ~ std.string.toString(one) ~ " " ~ f0);

    ArchiveMember am = new ArchiveMember();
    am.compressionMethod = 8;
    am.name = f0;
    //am.expandedData = cast(ubyte[]) f.read();
    am.expandedData = cast(ubyte[]) f.read();
    am.expandedSize = am.expandedData.length;
    FileData fd0 = GetFileInfo(f);
    long usedT0;
    if (fd0.creationTime > fd0.modifiedTime)
    {
      usedT0 = fd0.creationTime;
    }
    else
    {
      usedT0 = fd0.modifiedTime;
    }
    am.time = std.date.toDosFileTime(usedT0);
    zr.addMember(am);
    writefln(" ** PLEASE WAIT FOR FILE #" ~ std.string.toString(one) ~ " TO
BE ZIPPED ** ");

    one++;
  }
  writefln("finished all files...");
  writefln(" ** PLEASE WAIT FOR ZIP FILE TO BE CREATED ** ");
  std.file.write(zipFile, cast(byte[])zr.build());
  writefln(" ** " ~ std.string.toString(one) ~ " FILES ZIPPED ** ");
  writefln(zipFile ~ " created.");
  //return 1;
}

As you can see, the last two writelns do not get printed because of the Out
Of Memory.  Any help would be greatly appreciated.  Thanks.

josé

On 12/17/2012 10:01 PM, jicman wrote:
> Greetings!
>
> I have this program that zips a file and everything works perfectly, if the
> files are small enough. But, I am having to zip files that are getting more
> and more extreme in size and I am running out of memory. Here is the output
> when executing a small program to zip a folder:
>
> 0:36:59.76>zipafile
> Zipping file no 0 huge\0.pdf
> ** PLEASE WAIT FOR FILE #0 TO BE ZIPPED **
> Zipping file no 1 huge\1.exe
> ** PLEASE WAIT FOR FILE #1 TO BE ZIPPED **
> Zipping file no 2 huge\2.exe
> ** PLEASE WAIT FOR FILE #2 TO BE ZIPPED **
> Zipping file no 3 huge\updates\p.zip
> ** PLEASE WAIT FOR FILE #3 TO BE ZIPPED **
> Zipping file no 4 huge\updates\q.html
> ** PLEASE WAIT FOR FILE #4 TO BE ZIPPED **
> Zipping file no 5 huge\updates\z.exe
> ** PLEASE WAIT FOR FILE #5 TO BE ZIPPED **
> finished all files...
> ** PLEASE WAIT FOR ZIP FILE TO BE CREATED **
> Error: Out of memory
>
> 0:37:56.26>
>
> This is the program in question...
>
> import std.stdio;
> import std.file;
> import std.date;
> import std.zip;
> import std.zlib;
> import jic.libs.MyFile;
>
> int main(char[][] args)
> {
>
> char[] folder = r"c:\tmp\huge";
> ZipAFolder(folder);
> return 0;
> }
>
> void ZipAFolder(char[] zipFolder)
> {
> char[] fdname = std.path.getDirName(zipFolder);
> char[][] allfiles = std.file.listdir(zipFolder,"*");
> char[] zipFile = zipFolder ~ ".zip";
> int FilesCnt = 0;
>
> std.zip.ZipArchive zr;
> zr = new std.zip.ZipArchive();
> int one = 0;
> foreach(char[] f; allfiles)
> {
> char[] f0 = std.string.replace(f,fdname ~ "\\",""); // Filename
> writefln("Zipping file no " ~ std.string.toString(one) ~ " " ~ f0);
>
> ArchiveMember am = new ArchiveMember();
> am.compressionMethod = 8;
> am.name = f0;
> //am.expandedData = cast(ubyte[]) f.read();
> am.expandedData = cast(ubyte[]) f.read();
> am.expandedSize = am.expandedData.length;
> FileData fd0 = GetFileInfo(f);
> long usedT0;
> if (fd0.creationTime > fd0.modifiedTime)
> {
> usedT0 = fd0.creationTime;
> }
> else
> {
> usedT0 = fd0.modifiedTime;
> }
> am.time = std.date.toDosFileTime(usedT0);
> zr.addMember(am);
> writefln(" ** PLEASE WAIT FOR FILE #" ~ std.string.toString(one) ~ " TO
> BE ZIPPED ** ");
>
> one++;
> }
> writefln("finished all files...");
> writefln(" ** PLEASE WAIT FOR ZIP FILE TO BE CREATED ** ");
> std.file.write(zipFile, cast(byte[])zr.build());
> writefln(" ** " ~ std.string.toString(one) ~ " FILES ZIPPED ** ");
> writefln(zipFile ~ " created.");
> //return 1;
> }
>
> As you can see, the last two writelns do not get printed because of the Out
> Of Memory. Any help would be greatly appreciated. Thanks.
>
> josé

Is this a 32-bit program? It may be similar to the following issue:

  http://forum.dlang.org/thread/k95mf2$1aar$1@digitalmars.com?page=1

If so, you are advised to manage your own memory; do not rely on the garbage collector.

Ali

"Ali Çehreli"  wrote ...
> On 12/17/2012 10:01 PM, jicman wrote:
>> Greetings!
>>
>> I have this program that zips a file and everything works perfectly, if
>> the
>> files are small enough. But, I am having to zip files that are getting
>> more
>> and more extreme in size and I am running out of memory. Here is the
>> output
>> when executing a small program to zip a folder:
>>
>> 0:36:59.76>zipafile
>> Zipping file no 0 huge\0.pdf
>> ** PLEASE WAIT FOR FILE #0 TO BE ZIPPED **
>> Zipping file no 1 huge\1.exe
>> ** PLEASE WAIT FOR FILE #1 TO BE ZIPPED **
>> Zipping file no 2 huge\2.exe
>> ** PLEASE WAIT FOR FILE #2 TO BE ZIPPED **
>> Zipping file no 3 huge\updates\p.zip
>> ** PLEASE WAIT FOR FILE #3 TO BE ZIPPED **
>> Zipping file no 4 huge\updates\q.html
>> ** PLEASE WAIT FOR FILE #4 TO BE ZIPPED **
>> Zipping file no 5 huge\updates\z.exe
>> ** PLEASE WAIT FOR FILE #5 TO BE ZIPPED **
>> finished all files...
>> ** PLEASE WAIT FOR ZIP FILE TO BE CREATED **
>> Error: Out of memory
>>
>> 0:37:56.26>
>>
>> This is the program in question...
>>
>> import std.stdio;
>> import std.file;
>> import std.date;
>> import std.zip;
>> import std.zlib;
>> import jic.libs.MyFile;
>>
>> int main(char[][] args)
>> {
>>
>> char[] folder = r"c:\tmp\huge";
>> ZipAFolder(folder);
>> return 0;
>> }
>>
>> void ZipAFolder(char[] zipFolder)
>> {
>> char[] fdname = std.path.getDirName(zipFolder);
>> char[][] allfiles = std.file.listdir(zipFolder,"*");
>> char[] zipFile = zipFolder ~ ".zip";
>> int FilesCnt = 0;
>>
>> std.zip.ZipArchive zr;
>> zr = new std.zip.ZipArchive();
>> int one = 0;
>> foreach(char[] f; allfiles)
>> {
>> char[] f0 = std.string.replace(f,fdname ~ "\\",""); // Filename
>> writefln("Zipping file no " ~ std.string.toString(one) ~ " " ~ f0);
>>
>> ArchiveMember am = new ArchiveMember();
>> am.compressionMethod = 8;
>> am.name = f0;
>> //am.expandedData = cast(ubyte[]) f.read();
>> am.expandedData = cast(ubyte[]) f.read();
>> am.expandedSize = am.expandedData.length;
>> FileData fd0 = GetFileInfo(f);
>> long usedT0;
>> if (fd0.creationTime > fd0.modifiedTime)
>> {
>> usedT0 = fd0.creationTime;
>> }
>> else
>> {
>> usedT0 = fd0.modifiedTime;
>> }
>> am.time = std.date.toDosFileTime(usedT0);
>> zr.addMember(am);
>> writefln(" ** PLEASE WAIT FOR FILE #" ~ std.string.toString(one) ~ " TO
>> BE ZIPPED ** ");
>>
>> one++;
>> }
>> writefln("finished all files...");
>> writefln(" ** PLEASE WAIT FOR ZIP FILE TO BE CREATED ** ");
>> std.file.write(zipFile, cast(byte[])zr.build());
>> writefln(" ** " ~ std.string.toString(one) ~ " FILES ZIPPED ** ");
>> writefln(zipFile ~ " created.");
>> //return 1;
>> }
>>
>> As you can see, the last two writelns do not get printed because of the
>> Out
>> Of Memory. Any help would be greatly appreciated. Thanks.

> Is this a 32-bit program? It may be similar to the following issue:
Yes, this is a 32-bit program, but it is D1.0 vs. D2.0

>   http://forum.dlang.org/thread/k95mf2$1aar$1@digitalmars.com?page=1
>
> If so, you are advised to manage your own memory; do not rely on the garbage collector.

Thanks, Ali.

The problem is that the std.zip library does not allow me to break the write in pieces.  That is why I am asking for help.  How can I break this line,

std.file.write(zipFile, cast(byte[])zr.build());

to get and write chunks?  Any help would be greatly appreciated.  Thanks.

josé