View mode: basic / threaded / horizontal-split · Log in · Help
December 02, 2008
Re: Is "Out of Memory" a recoverable error?
== Quote from Russell Lewis (webmaster@villagersonline.com)'s article
> Walter Bright wrote:
> > I asked this over on stackoverflow.com to see what people using other
> > languages have to say, as well as the D community. The reason I ask is
> > to see if memory allocation can be allowed in functions marked "nothrow".
> >
> > http://stackoverflow.com/questions/333736/is-out-of-memory-a-recoverable-error
> It seems that D has (or rather, can have) a trivial solution to this
> problem.  Allow programs to register with the GC when they have memory
> which can be easily freed (caches and such).  Then you can make "out of
> memory" a non-recoverable error, since it only hits when we fail to
> recover enough.

Interesting idea.  It would certainly be easy enough to add to the GC.


Sean
December 02, 2008
Re: Is "Out of Memory" a recoverable error?
Reply to Russell,

> Walter Bright wrote:
> 
>> I asked this over on stackoverflow.com to see what people using other
>> languages have to say, as well as the D community. The reason I ask
>> is to see if memory allocation can be allowed in functions marked
>> "nothrow".
>> 
>> http://stackoverflow.com/questions/333736/is-out-of-memory-a-recovera
>> ble-error
>> 
> It seems that D has (or rather, can have) a trivial solution to this
> problem.  Allow programs to register with the GC when they have memory
> which can be easily freed (caches and such).  Then you can make "out
> of memory" a non-recoverable error, since it only hits when we fail to
> recover enough.
> 
>

Some sort of higher level of granularity would be nice, mostly an ordering 
hint re the different stages.

> 
> Seems to me that with this mechanism in place, we can treat
> out-of-memory as an unrecoverable error.
> 
> Thoughts?

vote += 0.5; // nice idea, not sure it would help *me* in any way though.
December 02, 2008
Re: Is "Out of Memory" a recoverable error?
One can get rid of OOM just by adding a couple of lines to the default malloc, that call custom handler (similar to onOutOfMemoryError handler) wich does any magic wanted, so OOM doesn't break nothrow functions, and if it's thrown, this means that recovery code has failed to recover and application exits. So nothrow protocol remains the same.
December 02, 2008
Re: Is "Out of Memory" a recoverable error?
== Quote from Kagamin (spam@here.lot)'s article
> One can get rid of OOM just by adding a couple of lines to the default malloc, that call custom handler
(similar to onOutOfMemoryError handler) wich does any magic wanted

Endless loop.


Sean
December 02, 2008
Re: Is "Out of Memory" a recoverable error?
Kagamin Wrote:

> One can get rid of OOM just by adding a couple of lines to the default malloc, that call custom handler (similar to onOutOfMemoryError handler) wich does any magic wanted, so OOM doesn't break nothrow functions, and if it's thrown, this means that recovery code has failed to recover and application exits. So nothrow protocol remains the same.

I think, preventing OOM is the only possible technique to make sure you don't interfere with the rest of the application.
December 03, 2008
Re: Is "Out of Memory" a recoverable error?
Don wrote:
> Strategy (1):
> Windows used to have a WM_COMPACTING message (maybe it still does) which 
> was sent when the system was running low on memory. In D, you could 
> imagine a similar sort of system callback, which is called when memory 
> is short -- it means, free some memory now, otherwise you'll get an out 
> of memory error.
> This is much simpler and more powerful than catching an 
> OutOfMemoryException, freeing some memory, and then repeating what you 
> were doing.

I like
December 03, 2008
Re: Is "Out of Memory" a recoverable error?
"Walter Bright" <newshound1@digitalmars.com> wrote in message 
news:gh38om$m7r$1@digitalmars.com...
>I asked this over on stackoverflow.com to see what people using other 
>languages have to say, as well as the D community. The reason I ask is to 
>see if memory allocation can be allowed in functions marked "nothrow".
>
> http://stackoverflow.com/questions/333736/is-out-of-memory-a-recoverable-error

I haven't looked at any of the other responses yet, but I'd have to say 
"sometimes". Clearly, getting an out of memory when trying to instantiate a 
trivial class is nonrecoverable (unless it's happening within a section of 
memory-intensive code that's allowed to fail, and the rest of the program 
does very little allocation), but getting an out of memory when trying to 
allocate a 1GB buffer for video processing is certainly recoverable.
December 03, 2008
Re: Is "Out of Memory" a recoverable error?
Russell Lewis, el  2 de diciembre a las 09:57 me escribiste:
> PRE-SCAN
> 
> Before the mark & sweep runs, *every one* of these callbacks is called.
> These are for things which the program can give up with very little
> cost, such as emptying free pools in allocators.  Since this happens
> before the scan, you do *not* have to use explicit "delete"s; you can
> just drop references as normal.  After all of these callbacks are
> called, the mark & sweep runs, and we hope that it will find some
> newly-discarded regions.

I think one could benefit from manually deleting stuff, if possible. This
way, space can be found before the collection, and you saved yourself some
collecting time. Manually deleting stuff could remain optional, no problem
with that, so if you have data that it's not that trivial to decide if
it's unused, you can just leave that work to the collector.

> Thoughts?

Looks really interesting. This could be a perfect complement to a way to
ask for memory when you expect the memory allocation to fail.

-- 
Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/
----------------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------------
People love to judge homeless guys.
Like, your giving him money he's just gonna waste it.
He's just gonna waste the money
Well, he lives in a box, what do you want him to do?
Save up and buy a wall unit?
Take a little run to the store for a throw rug and a CD rack?
He's homeless.
December 03, 2008
Re: Is "Out of Memory" a recoverable error?
Walter Bright, el  2 de diciembre a las 04:13 me escribiste:
> I asked this over on stackoverflow.com to see what people using other languages 
> have to say, as well as the D community. The reason I ask is to see if memory 
> allocation can be allowed in functions marked "nothrow".
> 
> http://stackoverflow.com/questions/333736/is-out-of-memory-a-recoverable-error

I think all the things said in this thread makes sense (adding callbacks
to the garbage collector, adding a way to ask for memory that don't throw
if the memory can't be allocated), but I think this don't cover all the
possible scenarios.

For example, I'm working on a softswitch (unfortunately no in D). Lets say
we have a really bad moment and all the subscribers want to talk at the
same time and we don't support that workload. Lets say our memory is
exhausted and a new call arrive. A new allocation is done somewhere deep
inside the call logic, so the PRE COLLECT callback is called. No memory
can be reclaimed, so the GC runs a collection. Still no memory. POST
COLLECT and CRISIS are called too without success. My softswitch is down,
I lost all the current calls. This is not good for business. What I really
wanted to do is to catch the memory error as shallow in the call logic as
possible and drop only that current call, leaving all the current
established calls intact.

So what can I do? Should I manually check each and every allocation in all
the call logic? I think that's unacceptable.

I think this scenario apply to each client-server application that needs
to stay alive even with high workloads, as the expense of dropping some
connection/client (web servers or web applications for example, as someone
said in stackoverflow.com).

-- 
Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/
----------------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------------
PADRES DENUNCIAN QUE SU HIJA SE ESCAPO CON UN PARAGUAYITO
	-- Crónica TV
December 03, 2008
Re: Is "Out of Memory" a recoverable error?
On Wed, 03 Dec 2008 08:34:43 -0500, Leandro Lucarella <llucax@gmail.com>  
wrote:

> Walter Bright, el  2 de diciembre a las 04:13 me escribiste:
>> I asked this over on stackoverflow.com to see what people using other  
>> languages
>> have to say, as well as the D community. The reason I ask is to see if  
>> memory
>> allocation can be allowed in functions marked "nothrow".
>>
>> http://stackoverflow.com/questions/333736/is-out-of-memory-a-recoverable-error
>
> I think all the things said in this thread makes sense (adding callbacks
> to the garbage collector, adding a way to ask for memory that don't throw
> if the memory can't be allocated), but I think this don't cover all the
> possible scenarios.
>
> For example, I'm working on a softswitch (unfortunately no in D). Lets  
> say
> we have a really bad moment and all the subscribers want to talk at the
> same time and we don't support that workload. Lets say our memory is
> exhausted and a new call arrive. A new allocation is done somewhere deep
> inside the call logic, so the PRE COLLECT callback is called. No memory
> can be reclaimed, so the GC runs a collection. Still no memory. POST
> COLLECT and CRISIS are called too without success. My softswitch is down,
> I lost all the current calls. This is not good for business. What I  
> really
> wanted to do is to catch the memory error as shallow in the call logic as
> possible and drop only that current call, leaving all the current
> established calls intact.
>
> So what can I do? Should I manually check each and every allocation in  
> all
> the call logic? I think that's unacceptable.
>
> I think this scenario apply to each client-server application that needs
> to stay alive even with high workloads, as the expense of dropping some
> connection/client (web servers or web applications for example, as  
> someone
> said in stackoverflow.com).
>

That's a good point/use case. However,
1) Many client-sever applications seperate each client into a logical  
thread of some kind. And inside this thread, out of memory is not  
recoverable.
2) One can still catch an error if need be.
1 2 3 4 5
Top | Discussion index | About this forum | D home