April 08, 2012
On Apr 7, 2012, at 12:37 PM, Rainer Schuetze <r.sagitario@gmx.de> wrote:

> 
> 
> On 4/7/2012 8:26 PM, Sean Kelly wrote:
>> On Apr 7, 2012, at 9:45 AM, Rainer Schuetze<r.sagitario@gmx.de>  wrote:
>> 
>>> 
>>> 
>>> On 4/6/2012 6:20 PM, deadalnix wrote:
>>>> Le 06/04/2012 18:07, Andrei Alexandrescu a écrit :
>>>>> A few more samples of people's perception of the two languages:
>>>>> 
>>>>> http://news.ycombinator.com/item?id=3805302
>>>>> 
>>>>> 
>>>>> Andrei
>>>> 
>>>> I did some measurement on that point for D lately : http://www.deadalnix.me/2012/03/05/impact-of-64bits-vs-32bits-when-using-non-precise-gc/
>>>> 
>>> 
>>> I studied the GC a bit more and noticed a possible issue:
>>> 
>>> - memory allocations are aligned up to a power of 2<= page size
>>> - the memory area beyond the actually requested size is left untouched when allocating
>>> - when the memory is collected, it is also untouched
>>> - the marking of references during collection does not know the requested size, so it scans the full memory block
>>> 
>>> Result: When a collected memory block is reused by a smaller allocation, there might still be false pointers in the unused area.
>>> 
>>> When I clear this data, my first impression is that it has improved the situation, but not enough. I'll have to create some non-interactive test to verify.
>> 
>> The unused area is typically zeroed out on allocation. Check GC.mallocNoSync..
> 
> That's where I added it ;-) But in fact it is in malloc/calloc and a few more places, though I haven't checked whether all the other calls of mallocNoSync handle the situation correctly. Almost all calls are to malloc anyway, so my patch doesn't change anything.

Is it not already there?  Maybe it was removed in all the GC optimizations?  I certainly remember a memset(0) in there.
April 08, 2012
On 4/7/12, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:
> 3. The ability to dispose of memory will disappear along with the delete keyword.

Pull this and hopefully that myth will come to an end: https://github.com/D-Programming-Language/d-programming-language.org/pull/112
April 08, 2012

On 4/8/2012 12:04 AM, Timon Gehr wrote:
> On 04/07/2012 04:43 PM, Rainer Schuetze wrote:
>>
>>
>> On 4/7/2012 8:24 AM, Dmitry Olshansky wrote:
>>> On 07.04.2012 2:08, Rainer Schuetze wrote:
>>>>
>>>>
>>>> On 4/6/2012 8:01 PM, Walter Bright wrote:
>>>>> On 4/6/2012 10:37 AM, Rainer Schuetze wrote:
>>>>>> I hope there is something wrong with my reasoning, and that you could
>>>>>> give me
>>>>>> some hints to avoid the memory bloat and the application stalls.
>>>>>
>>>>> A couple of things you can try (they are workarounds, not solutions):
>>>>>
>>>>> 1. Actively delete memory you no longer need, rather than relying on
>>>>> the
>>>>> gc to catch it. Yes, this is as unsafe as using C's free().
>>>>
>>>> Actually, having to deal with lifetime issues myself takes away the
>>>> biggest plus of the GC, so I am a bit reluctant to do this.
>>>>
>>>
>>> How about this:
>>> http://blog.thecybershadow.net/2010/07/15/data-d-unmanaged-memory-wrapper-for-d/
>>>
>>>
>>>
>>>
>>> Or you can wrap-up something similar along the same lines.
>>>
>>
>> Thanks for your and other's hints on reducing garbage collected memory,
>> but I find it hard to isolate larger blocks of memory for manual
>> management. Most of the structs and classes are rather small.
>>
>
> As you apparently just re-parse the whole source and throw the old AST
> away, wouldn't it be rather simple? You could just create a region
> allocator and free all the memory at once after the re-parse.

If you only use the syntax error highlighting feature that might work. As soon as semantic analysis kicks in modifications on a copy of the parse tree are done lazily, and as symbols get resolved, references into other trees are remembered. It can get rather involved to figure out when it is safe to delete a memory block manually.

The region allocator has the advantage that it does not need the alignment to the power of 2 for each allocation, so memory could be saved, too.

I guess, I'll try something along the line with some additional functions to avoid bad references after deletion.

>
>> I'm rather unhappy to sell D with the hint "Go back to manual memory
>> management if you need more than 64MB of memory and want your
>> application to be responsive."
>
> I think it is actually awesome that manual memory management is possible.
April 08, 2012
On 4/8/12 4:20 AM, Andrej Mitrovic wrote:
> On 4/7/12, Andrei Alexandrescu<SeeWebsiteForEmail@erdani.org>  wrote:
>> 3. The ability to dispose of memory will disappear along with the delete
>> keyword.
>
> Pull this and hopefully that myth will come to an end:
> https://github.com/D-Programming-Language/d-programming-language.org/pull/112

Done.

Andrei
April 08, 2012
On 4/6/2012 9:07 AM, Andrei Alexandrescu wrote:
> A few more samples of people's perception of the two languages:
>
> http://news.ycombinator.com/item?id=3805302


At least we don't have this issue: http://news.ycombinator.com/item?id=3814020

The D gc allocates smallish "chunks" as required, not one giant virtual arena.
April 08, 2012
On Sun, Apr 08, 2012 at 09:42:42AM -0500, Andrei Alexandrescu wrote:
> On 4/8/12 4:20 AM, Andrej Mitrovic wrote:
> >On 4/7/12, Andrei Alexandrescu<SeeWebsiteForEmail@erdani.org>  wrote:
> >>3. The ability to dispose of memory will disappear along with the delete keyword.
> >
> >Pull this and hopefully that myth will come to an end: https://github.com/D-Programming-Language/d-programming-language.org/pull/112
[...]

I have to say, that is one cool thing about having .init for all types. So we can cleanup after an object (close files, release resources, etc.) and still have the GC do whatever it needs to do, when it needs to. I love it!!


T

-- 
The day Microsoft makes something that doesn't suck is probably the day they start making vacuum cleaners... -- Slashdotter
April 08, 2012
On 4/8/12 12:21 PM, Walter Bright wrote:
> On 4/6/2012 9:07 AM, Andrei Alexandrescu wrote:
>> A few more samples of people's perception of the two languages:
>>
>> http://news.ycombinator.com/item?id=3805302
>
>
> At least we don't have this issue:
> http://news.ycombinator.com/item?id=3814020
>
> The D gc allocates smallish "chunks" as required, not one giant virtual
> arena.

On reddit, too: http://www.reddit.com/r/programming/comments/rzd19/do_not_use_go_for_32bit_development/

I find that behavior quite surprising because, unlike imprecise GC, is much easier to improve.

Anyhow, the recent discussions on Go clarify that we need to improve our collector's precision, and pronto. The only thing that didn't yet make the problem more painful in D is that D programs create a lot less garbage.


Andrei
April 08, 2012
On 8 April 2012 23:44, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>wrote:

> On 4/8/12 12:21 PM, Walter Bright wrote:
>
>> On 4/6/2012 9:07 AM, Andrei Alexandrescu wrote:
>>
>>> A few more samples of people's perception of the two languages:
>>>
>>> http://news.ycombinator.com/**item?id=3805302<http://news.ycombinator.com/item?id=3805302>
>>>
>>
>>
>> At least we don't have this issue: http://news.ycombinator.com/**item?id=3814020<http://news.ycombinator.com/item?id=3814020>
>>
>> The D gc allocates smallish "chunks" as required, not one giant virtual arena.
>>
>
> On reddit, too: http://www.reddit.com/r/**programming/comments/rzd19/do_** not_use_go_for_32bit_**development/<http://www.reddit.com/r/programming/comments/rzd19/do_not_use_go_for_32bit_development/>
>
> I find that behavior quite surprising because, unlike imprecise GC, is much easier to improve.
>
> Anyhow, the recent discussions on Go clarify that we need to improve our collector's precision, and pronto. The only thing that didn't yet make the problem more painful in D is that D programs create a lot less garbage.


What do you base that statistic on? I'm not arguing that fact, just that I
haven't seen any evidence one way or the other. What causes Go to create
significantly more garbage than D? Are there benchmarks or test cases I
should be aware of on the topic?
I'm quite curious to know more about this stuff, but I haven't found that
many comparisons between languages (in particular, C# would be of most
interest to me, since most game devs are most familiar with that)


April 08, 2012
On 4/8/2012 3:57 PM, Manu wrote:
> What do you base that statistic on? I'm not arguing that fact, just that I
> haven't seen any evidence one way or the other. What causes Go to create
> significantly more garbage than D? Are there benchmarks or test cases I should
> be aware of on the topic?

The first ycombinator reference is a person who didn't run out of memory using D. That implies far less pressure on the gc.

My understanding of Go is that when it does structural conformance, it builds some of the necessary data at runtime on the gc heap.

Anyhow, D has a lot of facilities for putting things on the stack rather than the heap, immutable data doesn't need to get copied, and slices allow lots of reuse of existing objects.
April 08, 2012
Walter:

> Anyhow, D has a lot of facilities for putting things on the stack rather than the heap,

In a system language heap allocations are often among the spots where the program is slower. So a good system language must offer good ways to use the stack as much as possible. D offers some ways to use the stack, but surely there is a good amount of space for improvements, some examples:

1) Some kind of optimization to avoid heap allocations for lines of code like this will improve the situation:
int[3] a = [10,20,30];

2) Some kind of (safer) stack-allocated variable-length arrays are handy (see Ada language too that uses stack-allocated arrays a lot):
http://d.puremagic.com/issues/show_bug.cgi?id=5348

3) Currently emplace has several problems and limits.

4) A compilation switch to list all points where a heap-allocetd closure is created will help avoid unwanted heap usage:
http://d.puremagic.com/issues/show_bug.cgi?id=5070

5 extra) Currently there are no ways to put an immutable associative array on the stack, or as global immutable variable :-) Many of my small associative arrays don't need to mutate after I have created them at the start of the program. Having a way to put them on the stack seems an interesting thing.

Bye,
bearophile