On 29.06.2013 06:33, Steven Schveighoffer wrote:
>
> On Jun 28, 2013, at 5:28 PM, Rainer Schuetze <r.sagitario@gmx.de>
> wrote:
>
>> On 28.06.2013 16:16, Steven Schveighoffer wrote:
> The associative array could just as easily make a struct out of that
> pair.  It so happens that when AA's were written, they were a purely
> runtime feature, and did not have the ability to "invent" typeinfo.
> The situation has changed -- AAs are now a template.
>

Unfortunately, both worlds still co-exist, and not always peacefully.

>> The current implementation "emplaces" the pointer bits at the
>> corresponding addresses, but this needs two additional calls into
>> the GC, resulting in far from optimal performance.
>
> You mean the current implementation of the precise GC or the AA?

The precise GC implementation that also patches AA implementation to supply type info through emplace.

>
> BTW, I want to stress again that we really should attempt to move the
> compiler away from calling runtime functions, and instead have it
> call template functions.  Such things will make life SOOO much easier
> for experimentation, inlining runtime calls, etc.  For example,
> having the rtInfo has opened several doors already that weren't
> already envisioned on its creation.  Imagine if all runtime calls
> were wrapped in a template, and one could simply update the template
> to experiment with new ideas, instead of hooking the runtime
> function, which is limited to whatever was expected when written.

That's what I am hoping for, too. But until the AA template implementation is working successfully I don't expect it to happen for other types or calls as well.

_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

Le 22-juin-2013 à 3:11, Walter Bright <walter@digitalmars.com> a écrit :

> Users really only want one gc.
> 
> [...]
> 
> We should pick precise collection and commit to it. And then add Leandro's concurrent collector on top!

I just want to point out that a concurrent collector that relies on "fork" will not work for iOS apps. Assuming someone would want to write one such app in D one day, the GC must still be able to work without "fork". It should work fine for sandboxed Mac apps though.

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca

_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

On 24.06.2013 20:29, Rainer Schuetze wrote:
> On 24.06.2013 11:19, Leandro Lucarella wrote:
>> I'll repeat what I said in my talk. I think the "rebuild" option is an
>> extremely bad idea.
>
> Especially in the transition to precise GC I would expect one or two
> hickups and everybody will tend to blame the changed GC. It should be
> easy to switch to non-precise GC to verify if the accusation is rectified.
>
>> I used "initialization-time" configurability in CDGC and worked pretty
>> well. It would be much more convenient to have a way to change GC
>> options at source code level too, but I had the same problem, I couldn't
>> find a way to do anything before the GC is initialized. I think being
>> able to configure runtime options via environment variables is an
>> extremely convenient option.
>
> I was thinking about environment variables aswell, but that's not so
> convenient for applications run by non-developers.
>
> One way to do it might be to put configuration variables into a module
> gc.config (it might still read environment variables to change
> defaults). This module is part of the druntime library. If the
> programmer then provides a different gc.config that fulfills all the
> link dependencies, on the link command line, it is preferred over the
> module from the library.
>
> One thing I'll do for sure is to convert versions(GC_PRECISE) to
> if(gc_precise), so it is easy to switch between a configuration variable
> and an enum that allows the optimizer to just remove the disabled parts.

I have pushed the respective changes to the repository, with reading an environment variable as the default. Exchanging the module on the linker command line also seems to work but expects the init function not to be inlined.


_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

On 28.06.2013 23:28, Rainer Schuetze wrote:
> There is still the problem of manually arranged data to be solved. E.g.
> the associative array combine node-list entry, key and value into a
> single allocation, and only the single type infos are available (if at
> all). Same problem as above: how to create a combined type info at runtime?
>
> The current implementation "emplaces" the pointer bits at the
> corresponding addresses, but this needs two additional calls into the
> GC, resulting in far from optimal performance.

I have added code to the AA implementation to create the combined TypeInfo at runtime (storing it inside the Impl struct). This removes the overhead of the additional calls to gc_emplace.

Using a slightly modified testgc3 from the dmd test suite I have made some benchmarks. It creates 10000 AA of type uint[uint] and adds 1000 entries to each. This is kind of a worst case for the GC because a lot of allocations are going on, but nothing can be collected up to the very end.

Doing some micro-optimizations in the GC I managed to get the precise GC (p_on) on par with or better than the existing GC (clean):

Desktop Core2Duo 6400 @ 2.13 GHz (Win32)

clean: peak mem = 182 MB,  17 collections in 10.816 s, overall 14.036 s
p_off: peak mem = 183 MB,  17 collections in 10.984 s, overall 14.327 s
p_on:  peak mem = 188 MB,  17 collections in  8.304 s, overall 11.940 s

p_off is the new implementation, but precise collection switched off. I believe the small overhead is due to most GC functions needing the type info as an additional parameter.

On my laptop, measurements are a bit less accurate due to the mobile CPU switching power state, but with disabling Turbo Boost, I get mostly reproducible results (best of 3 runs):

Mobile i7 2670QM @ 2.2 GHz (Win32)

clean: peak mem = 184 MB,  17 collections in  7.292 s, overall  9.648 s
p_off: peak mem = 185 MB,  17 collections in  7.734 s, overall 10.133 s
p_on:  peak mem = 190 MB,  17 collections in  6.743 s, overall  9.326 s

Mobile i7 2670QM @ 2.2 GHz (Win64)

clean: peak mem = 680 MB,  40 collections in 27.662 s, overall 30.643 s
p_off: peak mem = 680 MB,  40 collections in 29.731 s, overall 32.721 s
p_on:  peak mem = 691 MB,  40 collections in 27.793 s, overall 31.226 s

The 64-bit version needs almost 4 times as much memory because of some conservative assumptions for the alignment of the value in the (key,value) pair (unfortunately, the value type is not available in most functions). I'm not sure why the p_off version uses considerable more time collecting than the clean version. My guess is that some refactorings reduced the numbers of inlining going on.

When running the test in a loop, the GC should kick in, but really does not collect a lot in clean Win32 (most in clean win64). For 4 iterations, I get

win32_clean:  mem = 724 MB,  41 gcs in 99.473 s, overall 108.802 s
win32_p_on:   mem = 199 MB,  20 gcs in  7.329 s, overall 17.417 s

win64_clean:  mem = 698 MB,  43 gcs in 28.317 s, overall 39.364 s
win64_p_on:   mem = 708 MB,  43 gcs in 28.273 s, overall 41.142 s


_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

On Sat, Jun 29, 2013 at 10:34:32AM -0400, Michel Fortin wrote:
> Le 22-juin-2013 à 3:11, Walter Bright <walter@digitalmars.com> a écrit :
> 
> > Users really only want one gc.
> > 
> > [...]
> > 
> > We should pick precise collection and commit to it. And then add Leandro's concurrent collector on top!
> 
> I just want to point out that a concurrent collector that relies on "fork" will not work for iOS apps. Assuming someone would want to write one such app in D one day, the GC must still be able to work without "fork". It should work fine for sandboxed Mac apps though.

Yes, the collector will only enable the fork mechanism where it is available (or makes sense, I don't know if iOS supports fork but stuff break when you use it). Also forking should be user-configurable (all this is true for the current version in Tango).

-- 
Leandro Lucarella
Senior R&D Developer
Sociomantic Labs GmbH <http://www.sociomantic.com>
_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

On Jun 29, 2013, at 7:34 AM, Michel Fortin <michel.fortin@michelf.ca> wrote:

> Le 22-juin-2013 à 3:11, Walter Bright <walter@digitalmars.com> a écrit :
> 
>> Users really only want one gc.
>> 
>> [...]
>> 
>> We should pick precise collection and commit to it. And then add Leandro's concurrent collector on top!
> 
> I just want to point out that a concurrent collector that relies on "fork" will not work for iOS apps. Assuming someone would want to write one such app in D one day, the GC must still be able to work without "fork". It should work fine for sandboxed Mac apps though.

It might not work for Windows apps either, but Leandro's GC is a cleaner design than the existing GC.  If it can be brought in line with the existing GC in terms of features I'd prefer it as a base to build on.  It's also worth noting that Leandro's GC already supports precise GC by way of the old precise scanning proposal.
_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

On Wed, Jul 10, 2013 at 11:08:19AM -0700, Sean Kelly wrote:
> On Jun 29, 2013, at 7:34 AM, Michel Fortin <michel.fortin@michelf.ca> wrote:
> 
> > Le 22-juin-2013 à 3:11, Walter Bright <walter@digitalmars.com> a écrit :
> > 
> >> Users really only want one gc.
> >> 
> >> [...]
> >> 
> >> We should pick precise collection and commit to it. And then add Leandro's concurrent collector on top!
> > 
> > I just want to point out that a concurrent collector that relies on "fork" will not work for iOS apps. Assuming someone would want to write one such app in D one day, the GC must still be able to work without "fork". It should work fine for sandboxed Mac apps though.
> 
> It might not work for Windows apps either, but Leandro's GC is a cleaner design than the existing GC.  If it can be brought in line with the existing GC in terms of features I'd prefer it as a base to build on.  It's also worth noting that Leandro's GC already supports precise GC by way of the old precise scanning proposal.

Yes, but I would discard that because it never got in DMD and the way of storing the type information pointer at the end of the memory block didn't prove to be very successful. I have the feeling that Rainer's way of storing them in the pool instead (if I understood the talk correctly, I actually couldn't take a look at the code yet) should work better.

-- 
Leandro Lucarella
Senior R&D Developer
Sociomantic Labs GmbH <http://www.sociomantic.com>
_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

On Fri, Jun 21, 2013 at 11:37:45PM +0200, Rainer Schuetze wrote:
> On 21.06.2013 22:22, Steven Schveighoffer wrote:
> >>d2. Both dynamic and associative arrays currently allocate memory chunks and use them in a "non-standard" way that cannot be described by a simple TypeInfo. For example, dynamic arrays keep track of the allocated size of the array by placing it at the very end of allocations < 4k, but at the beginning for allocations >= 4k, moving the data to an offset of 16 in the latter case. Associative arrays combine hash-list-node, key and value into a single allocation, sometimes even with the value type unavailable.
> >
> >The dynamic array "allocated size" location and size is predictable, and can be determined whether it exists based on the APPENDABLE bit.  You should be able to correctly ignore it during a collection.  Or am I misunderstanding the reason you posted that?
> 
> 
> The size value itself is only a small issue, the larger one is the address of the array data moves depending on the size of the allocation, so the pointer info needs to be placed at some offset sometimes. My first implementation actually figured this out in the GC, but I think this leaks too much implementation details of the array into the GC. So I changed it to use gc_emplace instead.

Any reason not to use a more obvious name like setType() or similar?

-- 
Leandro Lucarella
Senior R&D Developer
Sociomantic Labs GmbH <http://www.sociomantic.com>
_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

On Fri, Jun 21, 2013 at 09:37:51PM +0200, Leandro Lucarella wrote:
> > Before creating a pull request, I'd like to hear opinions on whether this should be included, if other choices would be better and where it should be improved.
> 
> I'll read this e-mail in detail next week, and hopefully I'll take a look at the code too.

Finally I had some time to do this. I made some comments in the commit themselves in GitHub.

Is there any reason why not to do the review with a proper pull request? For some reason you prefer to keep this somehow more private for now?

-- 
Leandro Lucarella
Senior R&D Developer
Sociomantic Labs GmbH <http://www.sociomantic.com>
_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

On 07.08.2013 19:12, Leandro Lucarella wrote:
> On Fri, Jun 21, 2013 at 09:37:51PM +0200, Leandro Lucarella wrote:
>>> Before creating a pull request, I'd like to hear opinions on whether
>>> this should be included, if other choices would be better and where
>>> it should be improved.
>>
>> I'll read this e-mail in detail next week, and hopefully I'll take a
>> look at the code too.
>
> Finally I had some time to do this. I made some comments in the commit
> themselves in GitHub.
>

Thanks for the comments.

> Is there any reason why not to do the review with a proper pull request?
> For some reason you prefer to keep this somehow more private for now?
>

It currently even fails to compile the druntime unittests because of the reported problems with the RTInfo generation. May it is better to let it fail in public ;-)

I also wanted to implement a different version for comparison, that stores the TypeInfo pointer in the allocation. I have a first attempt but it is quite a bit slower as it has to do some dynamic casting during scanning because of the way arrays are implemented. Some optimizations are needed to make an unbiased comparison, but other stuff distracted from doing this.

I'm also hoping for a comment on the ambiguity of TypeInfo_Class. If implementing TypeInfo_Reference, what would TypeInfo_Reference.toString return? "C ref" as in Michel Fortin's implementation of tail const references? ;-)


_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime