I need to make some operations with all the files in a directory and subdirectories. Currently, I do it like this:

import std;

void DirIteration(string path) {
    try {
        foreach(entry; dirEntries(path, SpanMode.shallow, false)) { //SpanMode.shallow allows skip directories if any error happens
            if (entry.isFile && !entry.isSymlink)
                writeln(entry); //Or something instead of this
            if (entry.isDir)
                DirIteration(entry);
        }
    }
    catch (Throwable) {}
}

void main()
{
    DirIteration("C:\\Users\\angrypuppy\\MyDir");
}

But this method consumes a huge amount of memory (up to 4 GB and more). Is there a more appropriate way to walk directories recursively that does not consume a lot of memory?

On 1/14/21 6:46 PM, dog2002 wrote:
> I need to make some operations with all the files in a directory and subdirectories. Currently, I do it like this:
> 
> import std;
> 
> void DirIteration(string path) {
>      try {
>          foreach(entry; dirEntries(path, SpanMode.shallow, false)) { //SpanMode.shallow allows skip directories if any error happens
>              if (entry.isFile && !entry.isSymlink)
>                  writeln(entry); //Or something instead of this
>              if (entry.isDir)
>                  DirIteration(entry);
>          }
>      }
>      catch (Throwable) {}
> }
> 
> void main()
> {
>      DirIteration("C:\\Users\\angrypuppy\\MyDir");
> }
> 
> But this method consumes a huge amount of memory (up to 4 GB and more). Is there a more appropriate way to walk directories recursively that does not consume a lot of memory?

DirEntry is a struct. First of all I would try this:
```D
foreach(ref entry; dirEntries(path, SpanMode.shallow, false))
```

On 1/14/21 6:55 PM, drug wrote:
>>
>> But this method consumes a huge amount of memory (up to 4 GB and more). Is there a more appropriate way to walk directories recursively that does not consume a lot of memory?
> 
> DirEntry is a struct. First of all I would try this:
> ```D
> foreach(ref entry; dirEntries(path, SpanMode.shallow, false))
> ```

Does your directory just contain large amount of files?

On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote:
> On 1/14/21 6:55 PM, drug wrote:
>>>
>>> But this method consumes a huge amount of memory (up to 4 GB and more). Is there a more appropriate way to walk directories recursively that does not consume a lot of memory?
>> 
>> DirEntry is a struct. First of all I would try this:
>> ```D
>> foreach(ref entry; dirEntries(path, SpanMode.shallow, false))
>> ```
>
> Does your directory just contain large amount of files?

Yes. I forgot to add it in the original post.

On 1/14/21 7:06 PM, dog2002 wrote:
> On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote:
>> On 1/14/21 6:55 PM, drug wrote:
>>>>
>>>> But this method consumes a huge amount of memory (up to 4 GB and more). Is there a more appropriate way to walk directories recursively that does not consume a lot of memory?
>>>
>>> DirEntry is a struct. First of all I would try this:
>>> ```D
>>> foreach(ref entry; dirEntries(path, SpanMode.shallow, false))
>>> ```
>>
>> Does your directory just contain large amount of files?
> 
> Yes. I forgot to add it in the original post.

Does using `ref` changed anything?
Try following:
```
import std;

void DirIteration(ref DirEntry dir) {
    try {
        foreach(ref entry; dirEntries(dir, SpanMode.shallow, false)) { //SpanMode.shallow allows skip directories if any error happens
            if (entry.isFile && !entry.isSymlink)
                writeln(entry); //Or something instead of this
            if (entry.isDir)
                DirIteration(entry);
        }
    }
    catch (Throwable) {}
}

void main()
{
    auto de = DirEntry(".");
    DirIteration(de);
}
```

On Thursday, 14 January 2021 at 16:18:28 UTC, drug wrote:
> On 1/14/21 7:06 PM, dog2002 wrote:
>> On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote:
>>> [...]
>> 
>> Yes. I forgot to add it in the original post.
>
> Does using `ref` changed anything?
> Try following:
> ```
> import std;
>
> void DirIteration(ref DirEntry dir) {
>     try {
>         foreach(ref entry; dirEntries(dir, SpanMode.shallow, false)) { //SpanMode.shallow allows skip directories if any error happens
>             if (entry.isFile && !entry.isSymlink)
>                 writeln(entry); //Or something instead of this
>             if (entry.isDir)
>                 DirIteration(entry);
>         }
>     }
>     catch (Throwable) {}
> }
>
> void main()
> {
>     auto de = DirEntry(".");
>     DirIteration(de);
> }
> ```

No, it doesn't. Seems like memory can't clear.

On 1/14/21 7:30 PM, dog2002 wrote:
> On Thursday, 14 January 2021 at 16:18:28 UTC, drug wrote:
>> On 1/14/21 7:06 PM, dog2002 wrote:
>>> On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote:
>>>> [...]
>>>
>>> Yes. I forgot to add it in the original post.
>>
>> Does using `ref` changed anything?
>> Try following:
>> ```
>> import std;
>>
>> void DirIteration(ref DirEntry dir) {
>>     try {
>>         foreach(ref entry; dirEntries(dir, SpanMode.shallow, false)) { //SpanMode.shallow allows skip directories if any error happens
>>             if (entry.isFile && !entry.isSymlink)
>>                 writeln(entry); //Or something instead of this
>>             if (entry.isDir)
>>                 DirIteration(entry);
>>         }
>>     }
>>     catch (Throwable) {}
>> }
>>
>> void main()
>> {
>>     auto de = DirEntry(".");
>>     DirIteration(de);
>> }
>> ```
> 
> No, it doesn't. Seems like memory can't clear.

It is a recursion. Memory will be freed only after completion. Then I would try to get rid of recursion.

On 1/14/21 7:06 PM, dog2002 wrote:
> On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote:
>> On 1/14/21 6:55 PM, drug wrote:
>>>>
>>>> But this method consumes a huge amount of memory (up to 4 GB and more). Is there a more appropriate way to walk directories recursively that does not consume a lot of memory?
>>>
>>> DirEntry is a struct. First of all I would try this:
>>> ```D
>>> foreach(ref entry; dirEntries(path, SpanMode.shallow, false))
>>> ```
>>
>> Does your directory just contain large amount of files?
> 
> Yes. I forgot to add it in the original post.

How much files do you have? DirEntry size is 168 bytes only and dirEntry is lazy range so I'm curious what is the reason of huge memory consumption. Do you use Windows 32 bits between?

On Thursday, 14 January 2021 at 16:47:45 UTC, drug wrote:
> On 1/14/21 7:06 PM, dog2002 wrote:
>> On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote:
>>> On 1/14/21 6:55 PM, drug wrote:
>>>>>
>>>>> But this method consumes a huge amount of memory (up to 4 GB and more). Is there a more appropriate way to walk directories recursively that does not consume a lot of memory?
>>>>
>>>> DirEntry is a struct. First of all I would try this:
>>>> ```D
>>>> foreach(ref entry; dirEntries(path, SpanMode.shallow, false))
>>>> ```
>>>
>>> Does your directory just contain large amount of files?
>> 
>> Yes. I forgot to add it in the original post.
>
> How much files do you have? DirEntry size is 168 bytes only and dirEntry is lazy range so I'm curious what is the reason of huge memory consumption. Do you use Windows 32 bits between?

About 1000 large files.

I want to replace several first bytes in all the files, so I just copy the remaining bytes into a new file. Might this be the reason for high memory consumption? If so, is there a way not to copy the entire file, just delete first bytes and write the replaced bytes into the beginning of the file?

I use Windows x64.

On Thursday, 14 January 2021 at 20:23:37 UTC, dog2002 wrote:
> About 1000 large files.
>
> I want to replace several first bytes in all the files, so I just copy the remaining bytes into a new file. Might this be the reason for high memory consumption? If so, is there a way not to copy the entire file, just delete first bytes and write the replaced bytes into the beginning of the file?
>
> I use Windows x64.

What code are you using to copy the bytes? If you're reading the whole file into memory at once, that will consume a lot of memory.