I've created a bug report:
"Static data that must be scanned by the GC should be grouped"
http://d.puremagic.com/issues/show_bug.cgi?id=4650

For the lazy, here is a fragment of the bug report:
---
It would be nice if the compiler could group all the static that must really be scanned (programs static variables) together and make its limits available to the GC. It would be even nicer to leave static variables that have no pointers out of that group, and even much more nicer to create a pointer map like the one in the patch from bug 3463 to allow precise heap scanning. That way the only memory in the program that would have to be scanned conservatively will be the stack.
---

I wonder if this is even possible, and how much work would require in that case. I did almost no DMD hacking but if this is not too hard and/or if someone can give me some pointers on how to do it I'd like to give it a try. I suspect it could have a very significant positive impact in the GC performance.


Thanks.

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
Karma police
I've given all I can,
it's not enough,
I've given all I can
but we're still on the payroll.

Leandro Lucarella, el 15 de agosto a las 21:57 me escribiste:
> I've created a bug report:
> "Static data that must be scanned by the GC should be grouped"
> http://d.puremagic.com/issues/show_bug.cgi?id=4650
> 
> For the lazy, here is a fragment of the bug report:
> ---
> It would be nice if the compiler could group all the static that must really be scanned (programs static variables) together and make its limits available to the GC. It would be even nicer to leave static variables that have no pointers out of that group, and even much more nicer to create a pointer map like the one in the patch from bug 3463 to allow precise heap scanning. That way the only memory in the program that would have to be scanned conservatively will be the stack.
> ---
> 
> I wonder if this is even possible, and how much work would require in that case. I did almost no DMD hacking but if this is not too hard and/or if someone can give me some pointers on how to do it I'd like to give it a try. I suspect it could have a very significant positive impact in the GC performance.

I know this is probably really low priority for DMD, but any answer would be appreciated, even if it is something like "no idea how doable it is, it would take time to think about it and is low priority" :)

Thanks.

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
Y vos, me dijiste que soy, un abismo de silencio.
Ser?, porque vos no escuchas, que yo grito por dentro.

An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/dmd-internals/attachments/20100822/77015eca/attachment.html>

Walter Bright, el 22 de agosto a las 12:25 me escribiste:
>  I know this is probably really low priority for DMD, but any answer
>  would be appreciated, even if it is something like "no idea how doable
>  it is, it would take time to think about it and is low priority" :)
> 
> 
>    I don't think spending a lot of effort on the static data is worthwhile,
>    as very little of modern programs should consist of static mutable data.
> 
>    What's held me back from adding in precise scanning data for everything is
>    the large expansion of the memory footprint of a program to hold all this
>    data.

Thanks for the answer. My experiments show that static data is relevant.
Comparing 2 GC implementations, one with very few types (druntime
implementation) and another using a few more types, including a couple
of templates, the second is measurably slower. The types generates
a *lot* of rw static data.

Even when I agree that it might be too much to add type information to the static data, grouping only the static data that should be scanned (i.e. D-only global variables, and leaving out all the TypeInfo/ClassInfo) can make a big difference.

Would that be too hard?

Right now I'm under the serious dilemma of whether I should got back to
the more messy way of coding (with few types) or pay the price, which is
a very hard decision because I don't want to make the code uglier but
I don't want to have a slower GC either.

Thanks.

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
Se va a licitar un sistema de vuelos espaciales mendiante el cual, desde una
plataforma que quiz?s se instale en la provincia de C?rdoba. Esas naves
espaciales va a salir de la atm?sfera, va a remontar la estrat?sfera y desde
ah? elegir el lugar donde quieran ir de tal forma que en una hora y media
podamos desde Argentina estar en Jap?n, en Corea o en cualquier parte.
	-- Carlos Sa?l Menem (sic)

An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/dmd-internals/attachments/20100822/eaf852e0/attachment.html>

Walter Bright, el 22 de agosto a las 15:11 me escribiste:
>    Leandro Lucarella wrote:
> 
>  Walter Bright, el 22 de agosto a las 12:25 me escribiste:
> 
> 
>   I know this is probably really low priority for DMD, but any answer
>   would be appreciated, even if it is something like "no idea how doable
>   it is, it would take time to think about it and is low priority" :)
> 
> 
>     I don't think spending a lot of effort on the static data is worthwhile,
>     as very little of modern programs should consist of static mutable data.
> 
>     What's held me back from adding in precise scanning data for everything is
>     the large expansion of the memory footprint of a program to hold all this
>     data.
> 
> 
>  Thanks for the answer. My experiments show that static data is relevant.
>  Comparing 2 GC implementations, one with very few types (druntime
>  implementation) and another using a few more types, including a couple
>  of templates, the second is measurably slower. The types generates
>  a *lot* of rw static data.
> 
>  Even when I agree that it might be too much to add type information to
>  the static data, grouping only the static data that should be scanned
>  (i.e. D-only global variables, and leaving out all the
>  TypeInfo/ClassInfo) can make a big difference.
> 
>  Would that be too hard?
> 
> 
>    I can look into grouping it, but be aware I've had a lot of problems with
>    various linkers trying to get the grouping to work. The Mac linker in
>    particular behaves very unreliably and buggily with this, and its behavior
>    changes erratically with each new OSX release. I'd worry about it not
>    working correctly and the resulting app randomly crashing.

I will be willing to try it out in Linux if you do. Even a really ugly hack just to make some test would be very useful for me, so I can confirm or discard that the big static data is what is causing the problem.

Thanks.


-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
ASALTAN, GOLPEAN SALVAJEMENTE A ANCIANA Y LE COMEN LA PASTAFROLA.
	-- Cr?nica TV

I'm thinking - it's been the hallmark of druntime (vs. Phobos1) for the longest time that it has a configurable GC. How about this - let's make the precise GC a custom collector. This would allow us to (a) compare the existing GC with the proposed GC and (b) tune druntime's details with a real-world alternate GC.

Is that all possible?

Andrei

On 08/22/2010 04:57 PM, Leandro Lucarella wrote:
> Walter Bright, el 22 de agosto a las 12:25 me escribiste:
>>   I know this is probably really low priority for DMD, but any answer
>>   would be appreciated, even if it is something like "no idea how doable
>>   it is, it would take time to think about it and is low priority" :)
>>
>>
>>     I don't think spending a lot of effort on the static data is worthwhile,
>>     as very little of modern programs should consist of static mutable data.
>>
>>     What's held me back from adding in precise scanning data for everything is
>>     the large expansion of the memory footprint of a program to hold all this
>>     data.
>
> Thanks for the answer. My experiments show that static data is relevant.
> Comparing 2 GC implementations, one with very few types (druntime
> implementation) and another using a few more types, including a couple
> of templates, the second is measurably slower. The types generates
> a *lot* of rw static data.
>
> Even when I agree that it might be too much to add type information to the static data, grouping only the static data that should be scanned (i.e. D-only global variables, and leaving out all the TypeInfo/ClassInfo) can make a big difference.
>
> Would that be too hard?
>
> Right now I'm under the serious dilemma of whether I should got back to
> the more messy way of coding (with few types) or pay the price, which is
> a very hard decision because I don't want to make the code uglier but
> I don't want to have a slower GC either.
>
> Thanks.
>

Having a precise collector requires the compiler to emit the tables for it.

Andrei Alexandrescu wrote:
> I'm thinking - it's been the hallmark of druntime (vs. Phobos1) for the longest time that it has a configurable GC. How about this - let's make the precise GC a custom collector. This would allow us to (a) compare the existing GC with the proposed GC and (b) tune druntime's details with a real-world alternate GC.
>
> Is that all possible?
>
> Andrei
>

I see, thanks. Could that or at least the bulk of it be done through a reflection mechanism? That would have the compiler generate a modicum of info, leaving the library to generate the costly tables.

Andrei

On 08/22/2010 11:11 PM, Walter Bright wrote:
> Having a precise collector requires the compiler to emit the tables for it.
>
> Andrei Alexandrescu wrote:
>> I'm thinking - it's been the hallmark of druntime (vs. Phobos1) for the longest time that it has a configurable GC. How about this - let's make the precise GC a custom collector. This would allow us to (a) compare the existing GC with the proposed GC and (b) tune druntime's details with a real-world alternate GC.
>>
>> Is that all possible?
>>
>> Andrei
>>
> _______________________________________________
> dmd-internals mailing list
> dmd-internals at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals

>From what I understand, that was what the original precise scanning patch did.
Since the tables are static, having the compiler do it is a better idea.

But I think the "large footprint" might be somewhat of a strawman.  Yes, it will increase space, but I don't think it will be that much.  Consider that the precise scanning bits are static, and will be stored statically.  Have we run any tests to see how much space it actually adds?

-Steve



----- Original Message ----
> From: Andrei Alexandrescu <andrei at erdani.com>
> To: Discuss the internals of DMD <dmd-internals at puremagic.com>
> Sent: Mon, August 23, 2010 6:51:48 AM
> Subject: Re: [dmd-internals] Giving better static data limits to the GC
> 
> I see, thanks. Could that or at least the bulk of it be done through a reflection mechanism? That would have the compiler generate a modicum of info, leaving the library to generate the costly  tables.
> 
> Andrei
> 
> On 08/22/2010 11:11 PM, Walter Bright  wrote:
> > Having a precise collector requires the compiler to emit the  tables for it.
> >
> > Andrei Alexandrescu wrote:
> >> I'm  thinking - it's been the hallmark of druntime (vs. Phobos1) for
> >> the  longest time that it has a configurable GC. How about this - let's
> >>  make the precise GC a custom collector. This would allow us to (a)
> >>  compare the existing GC with the proposed GC and (b) tune druntime's
> >>  details with a real-world alternate GC.
> >>
> >> Is that all  possible?
> >>
> >> Andrei
> >>
> >  _______________________________________________
> > dmd-internals mailing  list
> > dmd-internals at puremagic.com
> > http://lists.puremagic.com/mailman/listinfo/dmd-internals
> _______________________________________________
> dmd-internals s mailing list
> dmd-internals at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>