Is it possible to make GC to free memory back to OS?

I'm asking because I'm writing a news feed notifier/reader, and my goal for the application is to be as much memory efficient as possible, as I intent to run it on my desktop 24/7.

In other words, I want my application to free any resources back to OS as soon as they no longer used. eg., it fetch a big document, process it store for later usage on disk cache, set a flag that new document available - then free all resources used for that task back to OS. Whatever peak memory usage will be, I want my application in idle state to have as small memory footprint as possible.

The 'high watermark' (if thats the name) approach in GC is not way to go in my case =(

There is not much info on GC on digitalmars site, so maybe someone know a workaround for my situation?

Thanks
--
serg.

Serg Kovrov wrote:
> Is it possible to make GC to free memory back to OS?
> 
> I'm asking because I'm writing a news feed notifier/reader, and my goal for the application is to be as much memory efficient as possible, as I intent to run it on my desktop 24/7.
> 
> In other words, I want my application to free any resources back to OS as soon as they no longer used. eg., it fetch a big document, process it store for later usage on disk cache, set a flag that new document available - then free all resources used for that task back to OS. Whatever peak memory usage will be, I want my application in idle state to have as small memory footprint as possible.
> 
> The 'high watermark' (if thats the name) approach in GC is not way to go in my case =(
> 
> There is not much info on GC on digitalmars site, so maybe someone know a workaround for my situation?

For a long time, even linux malloc()/free() would not return memory to the OS. Usually, memory is allocated from the OS using brk() or similar system calls and an application can only return memory from the end of the virtual address range the application has been allocated. In general, this requires a relocating/compacting GC, as even a single allocated byte at the top end of your virtual address space will prevent the OS from shrinking it.

But apart from needing much swap space, there are no draw backs of using much virtual memory. The memory you don't use (but exists in your virtual address range) will not map to physical memory and will not affect your computers performance.

/Oskar

* Oskar Linde:
> For a long time, even linux malloc()/free() would not return memory to the OS. Usually, memory is allocated from the OS using brk() or similar system calls and an application can only return memory from the end of the virtual address range the application has been allocated. In general, this requires a relocating/compacting GC, as even a single allocated byte at the top end of your virtual address space will prevent the OS from shrinking it.
> 
> But apart from needing much swap space, there are no draw backs of using much virtual memory. The memory you don't use (but exists in your virtual address range) will not map to physical memory and will not affect your computers performance.
> 
> /Oskar

I'm not sure, Oskar, as I am not an expert in that matter...

But... I'm not tried on linux but on Windows (my target OS is 2000/XP family so far) 'delete' do exactly what I want.

For example, in C++ (GCC/mingw) I could allocate with 'new' 50Mb, and see in task manager or performance monitor that my "private bytes" are increased about that size. When I 'delete' it, I see that "private bytes" count is dropped to the same level it was before. Nice.

As I understand "private bytes" are amount of memory my process uses exclusively and can not be used by anyone else. And that I define for myself as memory footprint - correct me if I wrong.

Same example in D show that GC greedily hold everything it get from OS.  And that is definitely what I do not want.

Oskar Linde wrote:
> But apart from needing much swap space, there are no draw backs of using much virtual memory. The memory you don't use (but exists in your virtual address range) will not map to physical memory and will not affect your computers performance.

That is not true in general:
| "For the attack to succeed, one needs to find a reliable way to
| force interesting kernel code to be paged out, then find that
| code inside a page file and modify it. And finally, the kernel
| needs to load that code (now modified) again into physical
| memory and execute it,"

http://www.darkreading.com/document.asp?doc_id=99780

---and for the current GC of D it is especially not true, because the 
sweeps of the GC do use the memory.

Karen Lanrap wrote:
> Oskar Linde wrote:
>> But apart from needing much swap space, there are no draw backs
>> of using much virtual memory. The memory you don't use (but
>> exists in your virtual address range) will not map to physical
>> memory and will not affect your computers performance.
> 
> That is not true in general:
> | "For the attack to succeed, one needs to find a reliable way to
> | force interesting kernel code to be paged out, then find that
> | code inside a page file and modify it. And finally, the kernel
> | needs to load that code (now modified) again into physical
> | memory and execute it,"
> 
> http://www.darkreading.com/document.asp?doc_id=99780
> 
> ---and for the current GC of D it is especially not true, because the sweeps of the GC do use the memory.

My point was that if the memory was collected by the GC, but not returned to the OS, the GC would not sweep it, and it would remain swapped out.

/Oskar

Oskar Linde wrote:

> My point was that if the memory was collected by the GC, but not returned to the OS, the GC would not sweep it, and it would remain swapped out.

I agree.

Karen Lanrap wrote:
> Oskar Linde wrote:
>> But apart from needing much swap space, there are no draw backs
>> of using much virtual memory. The memory you don't use (but
>> exists in your virtual address range) will not map to physical
>> memory and will not affect your computers performance.
> 
> That is not true in general:
> | "For the attack to succeed, one needs to find a reliable way to
> | force interesting kernel code to be paged out, then find that
> | code inside a page file and modify it. And finally, the kernel
> | needs to load that code (now modified) again into physical
> | memory and execute it,"

Wouldn't this also require the attacker to have root access to modify the page file?  Or are these always written with the same permissions as the program is executing in?


Sean

Oskar Linde wrote:
> Serg Kovrov wrote:
>> Is it possible to make GC to free memory back to OS?
>>
>> I'm asking because I'm writing a news feed notifier/reader, and my goal for the application is to be as much memory efficient as possible, as I intent to run it on my desktop 24/7.
>>
>> In other words, I want my application to free any resources back to OS as soon as they no longer used. eg., it fetch a big document, process it store for later usage on disk cache, set a flag that new document available - then free all resources used for that task back to OS. Whatever peak memory usage will be, I want my application in idle state to have as small memory footprint as possible.
>>
>> The 'high watermark' (if thats the name) approach in GC is not way to go in my case =(
>>
>> There is not much info on GC on digitalmars site, so maybe someone know a workaround for my situation?
> 
> For a long time, even linux malloc()/free() would not return memory to the OS. Usually, memory is allocated from the OS using brk() or similar system calls and an application can only return memory from the end of the virtual address range the application has been allocated. In general, this requires a relocating/compacting GC, as even a single allocated byte at the top end of your virtual address space will prevent the OS from shrinking it.
> 
> But apart from needing much swap space, there are no draw backs of using much virtual memory. The memory you don't use (but exists in your virtual address range) will not map to physical memory and will not affect your computers performance.

Just to clarify, the pages that are not in use will not map to physical memory.  This is another advantage of compacting collectors--they can empty out near-empty pages to avoid unnecessary swapping.


Sean