April 07, 2012
On 2012-04-07 17:41, Manu wrote:
> On 7 April 2012 17:03, Jacob Carlborg <doob@me.com <mailto:doob@me.com>>
> wrote:
>
>     On 2012-04-06 19:37, Rainer Schuetze wrote:
>
>
>         GC issues like this are currently blocking development of Visual
>         D (a
>         Win32 project): when just adding spaces to a file, parsing the
>         new file
>         every other second often needs 10 or more parsings until an
>         equal amount
>         of memory is collected compared to the allocated memory. AFAICT
>         Visual D
>         just keeps a reference to the root of the most recent AST of a
>         source file.
>
>         What's even worse: when the allocated memory gets larger (say >
>         200MB),
>         the garbage collection itself takes more than a second stalling the
>         application, which is a real pain if it happens while you are typing
>         source text (it does happen quite often).
>
>
>     Can you pause the GC when the user is typing? When you're finished
>     with the processing you can start it again.
>
>
> There's a bit of a problem there though, when you're coding, when are
> you NOT typing? :)
> I don't ever stop and sit there patiently for a few seconds for no reason.

It depends. I can do quite a lot of coding without typing. But yeah, I type a lot. It would basically turn on and off the GC, probably not good.

-- 
/Jacob Carlborg
April 07, 2012
On 07.04.2012 20:51, Manu wrote:
[snip]
>     I totally understand this sentiment, and unless GC improves by an
>     order of magnitude it is not going to work well with large to
>     medium-scale apps. Particularly long running ones, I once had been
>     running VisualD for about 16 hours straight (back in the days of
>     GSOC 2011) ;)
>
>
> Yeeesss.. I run VisualD for days at a time, and it just leaks memory
> until my computer chokes and crashes.
> It hovers between 1gb and 2gb usage under 'normal' usage for me, on a
> relatively small project (only 20-ish files).
> I am now in the habit if killing and restarting it regularly, but that's
> clearly not a good sign for real-world D apps...

I just turned off a couple of cool things like highlighting syntax errors. Seems to be fast and fluid afterwards.

-- 
Dmitry Olshansky
April 07, 2012
On 4/7/2012 7:43 AM, Rainer Schuetze wrote:
> I'm rather unhappy to sell D with the hint "Go back to manual memory management
> if you need more than 64MB of memory and want your application to be responsive."

Sure, that's why I said it was a workaround, not a solution.
April 07, 2012
On Apr 7, 2012, at 9:04 AM, Dmitry Olshansky <dmitry.olsh@gmail.com> wrote:
> 
> I totally understand this sentiment, and unless GC improves by an order of magnitude it is not going to work well with large to medium-scale apps. Particularly long running ones, I once had been running VisualD for about 16 hours straight (back in the days of GSOC 2011) ;)

I keep all my apps open pretty much indefinitely, and restart when things slow down too much. Safari, for example, tends to leak memory and needs to be restarted every few days on the outside.
April 07, 2012
On Apr 7, 2012, at 9:45 AM, Rainer Schuetze <r.sagitario@gmx.de> wrote:

> 
> 
> On 4/6/2012 6:20 PM, deadalnix wrote:
>> Le 06/04/2012 18:07, Andrei Alexandrescu a écrit :
>>> A few more samples of people's perception of the two languages:
>>> 
>>> http://news.ycombinator.com/item?id=3805302
>>> 
>>> 
>>> Andrei
>> 
>> I did some measurement on that point for D lately : http://www.deadalnix.me/2012/03/05/impact-of-64bits-vs-32bits-when-using-non-precise-gc/
>> 
> 
> I studied the GC a bit more and noticed a possible issue:
> 
> - memory allocations are aligned up to a power of 2 <= page size
> - the memory area beyond the actually requested size is left untouched when allocating
> - when the memory is collected, it is also untouched
> - the marking of references during collection does not know the requested size, so it scans the full memory block
> 
> Result: When a collected memory block is reused by a smaller allocation, there might still be false pointers in the unused area.
> 
> When I clear this data, my first impression is that it has improved the situation, but not enough. I'll have to create some non-interactive test to verify.

The unused area is typically zeroed out on allocation. Check GC.mallocNoSync.
April 07, 2012

On 4/7/2012 8:26 PM, Sean Kelly wrote:
> On Apr 7, 2012, at 9:45 AM, Rainer Schuetze<r.sagitario@gmx.de>  wrote:
>
>>
>>
>> On 4/6/2012 6:20 PM, deadalnix wrote:
>>> Le 06/04/2012 18:07, Andrei Alexandrescu a écrit :
>>>> A few more samples of people's perception of the two languages:
>>>>
>>>> http://news.ycombinator.com/item?id=3805302
>>>>
>>>>
>>>> Andrei
>>>
>>> I did some measurement on that point for D lately :
>>> http://www.deadalnix.me/2012/03/05/impact-of-64bits-vs-32bits-when-using-non-precise-gc/
>>>
>>
>> I studied the GC a bit more and noticed a possible issue:
>>
>> - memory allocations are aligned up to a power of 2<= page size
>> - the memory area beyond the actually requested size is left untouched when allocating
>> - when the memory is collected, it is also untouched
>> - the marking of references during collection does not know the requested size, so it scans the full memory block
>>
>> Result: When a collected memory block is reused by a smaller allocation, there might still be false pointers in the unused area.
>>
>> When I clear this data, my first impression is that it has improved the situation, but not enough. I'll have to create some non-interactive test to verify.
>
> The unused area is typically zeroed out on allocation. Check GC.mallocNoSync..

That's where I added it ;-) But in fact it is in malloc/calloc and a few more places, though I haven't checked whether all the other calls of mallocNoSync handle the situation correctly. Almost all calls are to malloc anyway, so my patch doesn't change anything.
April 07, 2012
On 04/06/2012 02:01 PM, Walter Bright wrote:
> On 4/6/2012 10:37 AM, Rainer Schuetze wrote:
>> I hope there is something wrong with my reasoning, and that you could
>> give me
>> some hints to avoid the memory bloat and the application stalls.
>
> A couple of things you can try (they are workarounds, not solutions):
>
> 1. Actively delete memory you no longer need, rather than relying on the
> gc to catch it. Yes, this is as unsafe as using C's free().
>

I keep reading that 'delete' is going to go away.  Is this even future-proof, or is code written in this fashion going to suffer a reckoning later on?

As an aside: I think this point generalizes to "avoid using the GC in cases where it is easy to do so".  D is very good at this due to having expressive structs, scope keyword, array slices, and a number of other features that synergize to track ownership/lifetime of memory without GC help.

> 2. Null out pointers & references when you are done with them. This
> helps reduce unwanted pinning of unused gc memory.
>
> 3. Minimize use of global memory, as that is a major source of source of
> roots.

April 07, 2012
On 4/7/12 4:26 PM, Chad J wrote:
> I keep reading that 'delete' is going to go away. Is this even
> future-proof, or is code written in this fashion going to suffer a
> reckoning later on?

Biggest confusions in the history of humankind:

1. Pyramids have been built by aliens;

2. Flying with a device heavier than air is impossible;

3. The ability to dispose of memory will disappear along with the delete keyword.


Andrei
April 07, 2012
On 04/07/2012 04:43 PM, Rainer Schuetze wrote:
>
>
> On 4/7/2012 8:24 AM, Dmitry Olshansky wrote:
>> On 07.04.2012 2:08, Rainer Schuetze wrote:
>>>
>>>
>>> On 4/6/2012 8:01 PM, Walter Bright wrote:
>>>> On 4/6/2012 10:37 AM, Rainer Schuetze wrote:
>>>>> I hope there is something wrong with my reasoning, and that you could
>>>>> give me
>>>>> some hints to avoid the memory bloat and the application stalls.
>>>>
>>>> A couple of things you can try (they are workarounds, not solutions):
>>>>
>>>> 1. Actively delete memory you no longer need, rather than relying on
>>>> the
>>>> gc to catch it. Yes, this is as unsafe as using C's free().
>>>
>>> Actually, having to deal with lifetime issues myself takes away the
>>> biggest plus of the GC, so I am a bit reluctant to do this.
>>>
>>
>> How about this:
>> http://blog.thecybershadow.net/2010/07/15/data-d-unmanaged-memory-wrapper-for-d/
>>
>>
>>
>> Or you can wrap-up something similar along the same lines.
>>
>
> Thanks for your and other's hints on reducing garbage collected memory,
> but I find it hard to isolate larger blocks of memory for manual
> management. Most of the structs and classes are rather small.
>

As you apparently just re-parse the whole source and throw the old AST away, wouldn't it be rather simple? You could just create a region allocator and free all the memory at once after the re-parse.

> I'm rather unhappy to sell D with the hint "Go back to manual memory
> management if you need more than 64MB of memory and want your
> application to be responsive."

I think it is actually awesome that manual memory management is possible.
April 07, 2012
On 04/07/2012 05:42 PM, Andrei Alexandrescu wrote:
> On 4/7/12 4:26 PM, Chad J wrote:
>> I keep reading that 'delete' is going to go away. Is this even
>> future-proof, or is code written in this fashion going to suffer a
>> reckoning later on?
>
> Biggest confusions in the history of humankind:
>
> 1. Pyramids have been built by aliens;
>
> 2. Flying with a device heavier than air is impossible;
>
> 3. The ability to dispose of memory will disappear along with the delete
> keyword.
>
>
> Andrei

Oh.  Whoops.

Thanks!