Greetings.

I am using,

15:32:35.63>dmd
Digital Mars D Compiler v1.046
Copyright (c) 1999-2009 by Digital Mars written by Walter Bright
Documentation: http://www.digitalmars.com/d/1.0/index.html

And I have a program that reads a file into UTF8 and does a series of string handling to create reports using an Associative Array of Arrays.  Then reads another file and does the same thing to each file and creates a report based on word usage, etc.  The problem is that the program is not releasing the memory.  Imagine this program:

//start
class TUCount
{
  int[char[]] File;
  char[][char[]] Target;
  int Count;
}

void ConsistencyCheck(char[] dir)
{
  TUCount[char[]] aTUs;
  char[][] allfiles = std.file.listdir(dir,"*.txt");
  aTUs = GrabUnits(allfiles);
  PrepareReport(aTUs);
}
TUCount[char[]] GrabUnits(char[][] allfiles)
{
  TUCount[char[]] aTUs;
  foreach (char[] f;allfiles)
  {
    char[] wText = "";
    wText = ReadFileData2UTF8(f, bom); //comes from another library and not in this file
                                       //<--Out of memory is happening in here...
    while (wText.length > 0)
    {
       // lots of some text handling and update aTUs base on text
    }
  }
}
void main
{
  char[] dir = r"C:\temp\LotsOfTextFiles";
  ConsistencyCheck(dir);
}
//end

The out of memory is happening in the ReadFileData2UTF function.  All that function does is to read the BOM and read the whole file into a variable and returns the UTF8 encoded string.  The problem is that apparently, it is reading the files and keeping that data there and never releasing it.  The taskmanager just keeps on growing and growing, etc.  I know that the aTUs content, which is being used to keep track of words, etc., is really low on memory usage, and it is not the cause of the huge amount of memory shown by the taskmanager.  I have 4G on a Win7 x32.  Any help would be appreciated.  Thanks.

josé

Depends on how you fill aTUs.

For example if you slice the original string, it will be preserved in memory. That's why parsers keep parsed substrings by duplicating them - this can result in smaller memory footprint.

On Tuesday, 7 April 2015 at 08:58:31 UTC, Kagamin wrote:
> Depends on how you fill aTUs.
Ok, I will bite... ;-)

I have the wText string which could be 20 mgs or so, I start finding pieces of data like this,

wText = wText[std.string.find(wText,"</ut>") + 5 .. $];

so, everything before </ut>, including it, will be thrown out, correct?  So, I continue like this, until I find a piece of the string that I want, and then, I fill the aTUs, like this,

aTUs = AddToTrackerRepeat(aTUs, source, fn, 1, target);

where:
 source is a part of the string wanted
 fn is the file name that the string was found
 1 is a count
 target is the other set of string wanted

And these are the other pieces missing:
  TUCount [char[]] AddToTrackerRepeat(TUCount[char[]] T, char[] tu, char[] f, int add, char[] target)
  {
    // target = target
    // f = filename
    // tu = translation unit
    // add = amount to be added
    if ((tu in T) == null)
    {
      T[tu] = new TUCount();
      T[tu].Count = 0;
      T[tu].File[f] = 0;
    }
    T[tu].Count += add;
    T[tu].File[f] += add;
    T[tu].Target[f ~ "\t" ~ std.string.toString(T[tu].File[f]) ] = target;
    return T;
  }

  class TUCount
  {
    int[char[]] File;
    char[][char[]] Target;
    int Count;
  }

On Tuesday, 7 April 2015 at 09:03:19 UTC, Kagamin wrote:
> For example if you slice the original string, it will be preserved in memory. That's why parsers keep parsed substrings by duplicating them - this can result in smaller memory footprint.

Hmmmm... Will you be able to give me an example of what is bad and then fix that bad to a good?  This may be my problem...

On Tuesday, 7 April 2015 at 15:28:21 UTC, jicman wrote:
> Hmmmm... Will you be able to give me an example of what is bad and then fix that bad to a good?  This may be my problem...

maybe
aTUs = AddToTrackerRepeat(aTUs, source.dup, fn, 1, target.dup);

On Friday, 10 April 2015 at 13:47:52 UTC, Kagamin wrote:
> On Tuesday, 7 April 2015 at 15:28:21 UTC, jicman wrote:
>> Hmmmm... Will you be able to give me an example of what is bad and then fix that bad to a good?  This may be my problem...
>
> maybe
> aTUs = AddToTrackerRepeat(aTUs, source.dup, fn, 1, target.dup);

This change causes an out of memory almost instantly.  Without this change, it takes longer to run out of memory.

Parsers unique duplicated strings via a name table:
string udup(string s, ref string[string] nameTable)
{
  if(s in nameTable)return nameTable[s];
  string s1=s.dup;
  nameTable[s1]=s1;
  return s1;
}

This way you avoid extra duplicates. You can also try to free file content manually when it's processed.

On Saturday, 11 April 2015 at 20:45:25 UTC, Kagamin wrote:
> Parsers unique duplicated strings via a name table:
> string udup(string s, ref string[string] nameTable)
> {
>   if(s in nameTable)return nameTable[s];
>   string s1=s.dup;
>   nameTable[s1]=s1;
>   return s1;
> }
>
> This way you avoid extra duplicates. You can also try to free file content manually when it's processed.

Hmmm...  Yes, definitely, that happens...  I will have to sit down and jump into out of memory abyss and how to handle it.  Thanks.

josé

On Saturday, 11 April 2015 at 20:45:25 UTC, Kagamin wrote:
> Parsers unique duplicated strings via a name table:
> string udup(string s, ref string[string] nameTable)
> {
>   if(s in nameTable)return nameTable[s];
>   string s1=s.dup;
>   nameTable[s1]=s1;
>   return s1;
> }
>
> This way you avoid extra duplicates. You can also try to free file content manually when it's processed.

This example helped so much.  Thanks.