On Thursday, 14 January 2021 at 22:28:19 UTC, Paul Backus wrote:
> On Thursday, 14 January 2021 at 20:23:37 UTC, dog2002 wrote:
>> About 1000 large files.
>>
>> I want to replace several first bytes in all the files, so I just copy the remaining bytes into a new file. Might this be the reason for high memory consumption? If so, is there a way not to copy the entire file, just delete first bytes and write the replaced bytes into the beginning of the file?
>>
>> I use Windows x64.
>
> What code are you using to copy the bytes? If you're reading the whole file into memory at once, that will consume a lot of memory.

void func(string inputFile, string outFile, uint chunk_size) {
	try {
		File _inputFile = File(inputFile, "r");
		File _outputFile = File(outFile, "w");
		
		ubyte[] tempBuffer = _inputFile.rawRead(new ubyte[](512));
	
                //doing some operations with the tempBuffer	

		_outputFile.rawWrite(tempBuffer);
		
		_inputFile.seek(tempBuffer.length, SEEK_SET);
		
		
		foreach(_buffer; _inputFile.byChunk(chunk_size)) {
			_outputFile.rawWrite(_buffer);
		}
		_inputFile.close();
		_outputFile.close();
	}
	catch (Throwable) {}

}

On Friday, 15 January 2021 at 06:15:06 UTC, dog2002 wrote:
>
> void func(string inputFile, string outFile, uint chunk_size) {
> 	try {
> 		File _inputFile = File(inputFile, "r");
> 		File _outputFile = File(outFile, "w");
> 		
> 		ubyte[] tempBuffer = _inputFile.rawRead(new ubyte[](512));
> 	
>                 //doing some operations with the tempBuffer	
>
> 		_outputFile.rawWrite(tempBuffer);
> 		
> 		_inputFile.seek(tempBuffer.length, SEEK_SET);
> 		
> 		
> 		foreach(_buffer; _inputFile.byChunk(chunk_size)) {
> 			_outputFile.rawWrite(_buffer);
> 		}
> 		_inputFile.close();
> 		_outputFile.close();
> 	}
> 	catch (Throwable) {}
>
> }

You can save a little bit of memory here by allocating tempBuffer on the stack:

    ubyte[512] tempBuffer;
    _inputFile.rawRead(tempBuffer[]); // note the explicit []

    // ...

    _outputFile.rawWrite(tempBuffer[]);

However, those allocations alone shouldn't be enough to get you to 4GB+, so the real issue is probably elsewhere.

On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus wrote:
>
> You can save a little bit of memory here by allocating tempBuffer on the stack:
>
>     ubyte[512] tempBuffer;
>     _inputFile.rawRead(tempBuffer[]); // note the explicit []

I made a mistake; this should be:

    ubyte[512] tempArray;
    ubyte[] tempBuffer = _inputFile.rawRead(tempArray[]);

...with the rest the same as your original version.

On Friday, 15 January 2021 at 06:33:55 UTC, Paul Backus wrote:
> On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus wrote:
>>
>> You can save a little bit of memory here by allocating tempBuffer on the stack:
>>
>>     ubyte[512] tempBuffer;
>>     _inputFile.rawRead(tempBuffer[]); // note the explicit []
>
> I made a mistake; this should be:
>
>     ubyte[512] tempArray;
>     ubyte[] tempBuffer = _inputFile.rawRead(tempArray[]);
>
> ...with the rest the same as your original version.

Thank you so much! It saves a lot of memory!

And one last question: why the application crashes, if I allocate 1 MB array?

>ubyte[1024000] tempBuffer;

On Friday, 15 January 2021 at 06:56:36 UTC, dog2002 wrote:
> On Friday, 15 January 2021 at 06:33:55 UTC, Paul Backus wrote:
>> On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus wrote:
>>>
>>> You can save a little bit of memory here by allocating tempBuffer on the stack:
>>>
>>>     ubyte[512] tempBuffer;
>>>     _inputFile.rawRead(tempBuffer[]); // note the explicit []
>>
>> I made a mistake; this should be:
>>
>>     ubyte[512] tempArray;
>>     ubyte[] tempBuffer = _inputFile.rawRead(tempArray[]);
>>
>> ...with the rest the same as your original version.
>
> Thank you so much! It saves a lot of memory!
>
> And one last question: why the application crashes, if I allocate 1 MB array?
>
>>ubyte[1024000] tempBuffer;

Solved:

ubyte[] tempBuffer = new ubyte[1024000];

On Fri, Jan 15, 2021 at 8:00 AM dog2002 via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote:

> On Friday, 15 January 2021 at 06:33:55 UTC, Paul Backus wrote:
> > On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus wrote:
> >>
> >> You can save a little bit of memory here by allocating tempBuffer on the stack:
> >>
> >>     ubyte[512] tempBuffer;
> >>     _inputFile.rawRead(tempBuffer[]); // note the explicit []
> >
> > I made a mistake; this should be:
> >
> >     ubyte[512] tempArray;
> >     ubyte[] tempBuffer = _inputFile.rawRead(tempArray[]);
> >
> > ...with the rest the same as your original version.
>
> Thank you so much! It saves a lot of memory!
>
> And one last question: why the application crashes, if I allocate 1 MB array?
>
> >ubyte[1024000] tempBuffer;
>

Because of stack overflow

On Fri, Jan 15, 2021 at 8:20 AM dog2002 via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote:

> On Friday, 15 January 2021 at 06:56:36 UTC, dog2002 wrote:
> > On Friday, 15 January 2021 at 06:33:55 UTC, Paul Backus wrote:
> >> On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus wrote:
> >>>
> >>> You can save a little bit of memory here by allocating tempBuffer on the stack:
> >>>
> >>>     ubyte[512] tempBuffer;
> >>>     _inputFile.rawRead(tempBuffer[]); // note the explicit []
> >>
> >> I made a mistake; this should be:
> >>
> >>     ubyte[512] tempArray;
> >>     ubyte[] tempBuffer = _inputFile.rawRead(tempArray[]);
> >>
> >> ...with the rest the same as your original version.
> >
> > Thank you so much! It saves a lot of memory!
> >
> > And one last question: why the application crashes, if I allocate 1 MB array?
> >
> >>ubyte[1024000] tempBuffer;
>
> Solved:
>
> ubyte[] tempBuffer = new ubyte[1024000];
>

You can still use ubyte[1024000] tempBuffer; but you have to place it
somewhere outside recursion or use a static
static ubyte[1024000] tempBuffer;

On Friday, 15 January 2021 at 07:16:21 UTC, Daniel Kozak wrote:
> On Fri, Jan 15, 2021 at 8:00 AM dog2002 via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote:
>
>> On Friday, 15 January 2021 at 06:33:55 UTC, Paul Backus wrote:
>> > On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus wrote:
>> >>
>> >> You can save a little bit of memory here by allocating tempBuffer on the stack:
>> >>
>> >>     ubyte[512] tempBuffer;
>> >>     _inputFile.rawRead(tempBuffer[]); // note the explicit []
>> >
>> > I made a mistake; this should be:
>> >
>> >     ubyte[512] tempArray;
>> >     ubyte[] tempBuffer = _inputFile.rawRead(tempArray[]);
>> >
>> > ...with the rest the same as your original version.
>>
>> Thank you so much! It saves a lot of memory!
>>
>> And one last question: why the application crashes, if I allocate 1 MB array?
>>
>> >ubyte[1024000] tempBuffer;
>>
>
> Because of stack overflow

A compiler parameter can be used to increase the maximum stack size
"dflags": ["-L/STACK:1500000000"]

or recursion can be somehow emulated using heap memory. Here is my "fake" recursion:
    // wins is a range

    auto stack = wins.save;
    while(!stack.empty){
        immutable n = stack.length - 1;
        auto window = stack[n];

        doSomeThingforEachRecursiveElement(window)

        stack.popBack;
        if(window.children.length){
            foreach (ref child; window.children)
                stack.pushBack(child);
        }
    }
    stack.free;

On Friday, 15 January 2021 at 06:15:06 UTC, dog2002 wrote:
> On Thursday, 14 January 2021 at 22:28:19 UTC, Paul Backus wrote:
>> On Thursday, 14 January 2021 at 20:23:37 UTC, dog2002 wrote:
>>> [...]
>>
>> What code are you using to copy the bytes? If you're reading the whole file into memory at once, that will consume a lot of memory.
>
> void func(string inputFile, string outFile, uint chunk_size) {
> 	try {
> 		File _inputFile = File(inputFile, "r");
> 		File _outputFile = File(outFile, "w");
> 		
> 		ubyte[] tempBuffer = _inputFile.rawRead(new ubyte[](512));
> 	
>                 //doing some operations with the tempBuffer	
>
> 		_outputFile.rawWrite(tempBuffer);
> 		
> 		_inputFile.seek(tempBuffer.length, SEEK_SET);
> 		
> 		
> 		foreach(_buffer; _inputFile.byChunk(chunk_size)) {
> 			_outputFile.rawWrite(_buffer);
> 		}
> 		_inputFile.close();
> 		_outputFile.close();
> 	}
> 	catch (Throwable) {}
>
> }

Okay, the reason is incredibly stupid: using WinMain instead of main causes high memory usage. I don't know why, I use the same code. If I replace WinMain with main, the memory consumption is about 6 MB.

On Fri, Jan 15, 2021 at 10:30 AM dog2002 via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote:

> ...
> Okay, the reason is incredibly stupid: using WinMain instead of
> main causes high memory usage. I don't know why, I use the same
> code. If I replace WinMain with main, the memory consumption is
> about 6 MB.
>

https://wiki.dlang.org/D_for_Win32