On Saturday, 5 January 2019 at 22:05:19 UTC, Manu wrote:
> I'm somewhere between a light GC user and a @nogc user, and I don't really know much about where we're at, or much about start-of-the-art GC in general.
>
> I read comments like this: https://www.reddit.com/r/programming/comments/acsg61/version_20840_of_dmd_the_d_reference_compiler_has/edbhgzi
>
> """
> And so we do nothing and D continues to languish as every thread is
> filled with discussion about a poor implementation of a sorely
> outdated GC? Other languages in this domain get by just fine - Go and
> Nim to name some off the top of my head (with the full disclosure that
> the former isn't quite D performance and Nim uses a thread-local GC).
> There are C / C++ libraries that provide a better GC. D's tradeoff is
> basically having one of the worst GCs of any language I've used this
> century to avoid a write barrier that causes a, what, 5%
> nigh-deterministic overall reduction in performance (assuming we use
> fast barriers out of a cycle).
>
> And if a reasonably performant GC is fundamentally incompatible and
> you think it will always be this way, then maybe D as a language needs
> to make serious re-evaluations? D keeps saying "Don't Fear the
> Reaper", but D is one of the only languages where I actually do.
> """
>
> How much truth is in here?
> What is this business about write barriers making the GC fast? Is that
> actually fact, or are there other hurdles?
>
> Is the claim that write-barriers generally slow your runtime
> performance significant? And if it is, is it possible/reasonable to
> control the places where they are emit?
> Can @nogc be used to inhibit write barriers in code that we know
> doesn't interact with the GC, such that we have control over that loss
> of perf?
>
> It's obvious that the quality of the GC implementation is a mental barrier to entry for many... and D has a GC, which is *attractive* to some users. Despite the fact that I don't care about the GC much personally, we do need to care about this as a group, and nobody seems to be making substantial progress.
>
> Is progress possible, or is the hard reality that the language is just designed such to be resistant to a quality GC, while the ecosystem sadly tends to rely on it?
>
> Where's the ARC stuff? What happened to opAddRef/opRelease?

There is this article [1] which covers some of the details and opportunities to get the D GC in a better shape.

For a while improvements on GC front were slow, I see that lately Rainer has submitted some PRs [2] for getting various GC issues sorted. My favorite one is the new precise GC [3].

I think a big improvement would be the use of thread local heaps and GC. But this requires a rethink of `shared`, which would be nice to be done anyhow.

1 - https://olshansky.me/gc/runtime/dlang/2017/06/14/inside-d-gc.html
2 - https://github.com/dlang/druntime/commits?author=rainers&since=2018-12-1&until=2019-01-07
3 - https://github.com/dlang/druntime/pull/2418

On Monday, 7 January 2019 at 06:52:40 UTC, H. S. Teoh wrote:
> I've experienced a similar effect in Firefox, and though I cannot say for sure it isn't a GC problem, I notice that it causes a long spike of intensive I/O, and appears to be correlated with occasional segfaults and signs of memory corruption / memory leak, and generally happens only after significant usage over a prolonged timeframe, generally ending up in a state of extreme memory usage (several GBs in resident set size for just a small number of persistent tabs) that reset to more reasonable levels upon restarting and restoring exactly the same tabs.

Firefox/Thunderbird are written mostly in C/C++/Rust. SpiderMonkey (the JS engine) does have a GC, but that one is incremental and generational and limited to JS memory. So I think it's unlikely that GC is the culprit.

There are any number of reasons why Firefox/Thunderbird might freeze, from synchronous JavaScript to inefficient or misplaced [1] SQLite queries.

[1] E.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1266829

On Sunday, 6 January 2019 at 08:52:53 UTC, Walter Bright wrote:
> On 1/5/2019 11:58 PM, Manu wrote:
>> If we can help the GC in common code, but still have the expressive
>> power we need to elide that extra work where we know it's not
>> necessary, then it might be reasonable to consider.
>
> If we could do that, we wouldn't need GC, the compiler could figure out the last use of each pointer. With all the research into GCs, nobody has ever figured that out.

Dont think of it as all or nothing. With dip1000 compiler knows the life time of some objects and if they have a reference that outlives objects scope. With this information compiler can replace some GC allocations with stack or specialized malloc allocations with deallocation at the end of the scope. This simple addition to compiler could increase performance of GC code by a lot in poorly written code.

With some mentoring I could even implement it myself. It doesnt sound that hard (famous last words) since lifetime tracking is already implemented

On 1/5/19 6:12 PM, Neia Neutuladh wrote:
> On Sat, 05 Jan 2019 14:05:19 -0800, Manu wrote:
>> Is progress possible, or is the hard reality that the language is just
>> designed such to be resistant to a quality GC, while the ecosystem sadly
>> tends to rely on it?
> 
> Three things about D make it harder to make a good GC for it:
> * unaligned pointers

Unaligned pointers are not scanned, and so this is not a problem. As far as the GC is concerned, they aren't pointers.

Perhaps you meant interior pointers?

> * unions

Unions and void * (or essentially untyped blocks) are the biggest problems. But depending on what we want to implement, we may have solutions for that.

It's difficult to paint with broad brushes here. There are certain problems that prevent certain solutions. But, for instance, a precise GC is getting traction as we speak: https://github.com/dlang/druntime/pull/2418

But precise GC is not necessarily better performing. And it's still "stop-the-world".

I think the biggest improvement for D's GC could be the thread-local pools (and avoid stop-the-world for those), but since casting shared and local data around is valid, I don't know how it can be implemented without language changes.

There are always extra heuristics and algorithmic improvements that people can analyze in the current GC as well.

-Steve

On Saturday, 5 January 2019 at 22:05:19 UTC, Manu wrote:
> poor implementation of a sorely outdated GC?
The whole r/programming needs to be garbage collected. I cant find a source but if I recall correctly mark and sweep GC has the highest program throughput of all GCs.

> What is this business about write barriers making the GC fast?

In Java and other managed languages common code has lots of small short lived objects and not a lot of pointer modifications so adding write barriers can get you concurrent GC for almost free. For a programmer like you that can turn heap allocations to stack, use scope(exit) or similar for simple allocations and leave difficult allocations to GC you wont benefit from them.


> It's obvious that the quality of the GC implementation is a mental barrier to entry for many...<...> we do need to care about this as a group, and nobody seems to be making substantial progress.

For things to change we need to change minds. I have watched many debates in religion, climate change and politics and saw just how hard it is to change some people opinions. The best hope is to target people in the middle that dont have strong opinions one way or another and repeatedly expose to information that shows that GC is not a problem.

Its not correct to say that people in this group dont care its just that everything needs work and this is not the only problem that D have. I have idea for D blog that would some what address this problem but I know myself and I know that there is low chance for this idea to materialize. I even picked title - "Why D with GC is faster than your C/C++ code" TL;DR memory management

> Is progress possible, or is the hard reality that the language is just designed such to be resistant to a quality GC

With dip1000 in principle we can reduce overhead of GC by replacing GC allocations to stack or malloc making generational GC necessary.

On Monday, 7 January 2019 at 06:52:40 UTC, H. S. Teoh wrote:
> I've experienced a similar effect in Firefox, and though I cannot say for sure it isn't a GC problem, I notice that it causes a long spike of intensive I/O, and appears to be correlated with occasional segfaults and signs of memory corruption / memory leak, and generally happens only after significant usage over a prolonged timeframe, generally ending up in a state of extreme memory usage (several GBs in resident set size for just a small number of persistent tabs) that reset to more reasonable levels upon restarting and restoring exactly the same tabs.
>
>
> T

this sound a lot like memory fragmentation
The Curse of External Fragmentation: Relocate or Bust!
http://ithare.com/the-curse-of-external-fragmentation-relocate-or-bust/

On Monday, 7 January 2019 at 15:54:32 UTC, welkam wrote:
> For things to change we need to change minds. I have watched many debates in religion, climate change and politics and saw just how hard it is to change some people opinions. The best hope is to target people in the middle that dont have strong opinions one way or another and repeatedly expose to information that shows that GC is not a problem.

I think for many people (especially C++ folks which seem to be the main target of D leadership) the mere existence of GC is enough to turn them off. Here's a quote I found on Rust subreddit today:

"I remember getting excited about D in 2002 exactly because it was supposed to be 'C++-improved' and then learning that D had GC which made it less 'C++-improved' and more 'Java-improved'. I switched to Rust exactly because I find it 'C++-improved'."

The mere existence of GC instantly puts D into application programming language rather than systems programming language for most people. Are they correct? Debatable. However the mental barrier is there and it's hard to get through with a nice message.

On Monday, 7 January 2019 at 17:18:49 UTC, JN wrote:
> On Monday, 7 January 2019 at 15:54:32 UTC, welkam wrote:
>> For things to change we need to change minds. I have watched many debates in religion, climate change and politics and saw just how hard it is to change some people opinions. The best hope is to target people in the middle that dont have strong opinions one way or another and repeatedly expose to information that shows that GC is not a problem.
>
> I think for many people (especially C++ folks which seem to be the main target of D leadership) the mere existence of GC is enough to turn them off. Here's a quote I found on Rust subreddit today:
>
> "I remember getting excited about D in 2002 exactly because it was supposed to be 'C++-improved' and then learning that D had GC which made it less 'C++-improved' and more 'Java-improved'. I switched to Rust exactly because I find it 'C++-improved'."
>
> The mere existence of GC instantly puts D into application programming language rather than systems programming language for most people. Are they correct? Debatable. However the mental barrier is there and it's hard to get through with a nice message.

Your sampling is bad. You pay attention to those who commented and not the silent majority. People who dont have strong opinions and open minded wont comment with such strong opinion on reddit threads discoursing language merits. They just read them and thats the people you can convince with solid arguments.

On Mon, Jan 07, 2019 at 05:18:49PM +0000, JN via Digitalmars-d wrote: [...]
> I think for many people (especially C++ folks which seem to be the main target of D leadership) the mere existence of GC is enough to turn them off.

Yep. GC phobia is a common malady among C/C++ folks (I used to be one of them).  I wrote D off the first time I found it because I was suffering from a particularly acute case of GC phobia, mostly colored by the bad impressions of the early versions of Java.

It took me a while to come around.  Reading this helped a little:

	https://dlang.org/spec/garbage.html

but it didn't convince me, only made me cautiously willing to at least try out D a little to see for myself.  I think I wasn't even willing to try D until I read that I could optionally still use malloc/free if I really wanted to, or if the GC turned out to be a problem.  Clutching onto that option was an important crutch for my initial foray into D.

It wasn't until I actually wrote code that let the GC do its job, that I was finally able to see, in retrospect, just how much time I wasted basically reimplementing my own version of GC (manually, and rather poorly!) and debugging the associated tricky pointer code -- all of which was unnecessary when there's a GC.

Furthermore, having a GC enabled a cleanness in my internal APIs that manually-managed memory did not allow -- it eliminated memory management from being an integral aspect of my APIs and freed up the design space (and my mental concentration!) to actually focus on the problem domain instead of being bogged down with memory management details.

The result was:

(1) my code quality was better from having eliminated an entire class of pointer bugs and from my being able to focus more on the problem domain rather than fiddle with memory management at every turn; and

(2) it had better API that directly addressed the problem domain, uncluttered by dirty memory management specifics;

(3) I could achieve the above in a much shorter time (and with much less
effort) than without the GC, since I eliminated the big time sink of
manual memory management.

This was a clear win for me.  So eventually I convinced myself. :-P


The other important thing I had to get over was the oft-cited performance concerns -- which in my case (and I surmise in the case of many others in the C/C++ camp) was colored by the bad impressions of early versions of Java.  It took me a while to convince myself that the chains of `delete`s in the C++ dtor of a large recursive structure basically was tantamount to a GC pause, just scheduled differently. Well, OK, it's not *exactly* the same thing, but it dispelled the wrong notion that malloc/free (or C++ new/delete) came "for free".  You still have to do roughly the same amount of "work" or bookkeeping either way.

Of course, the cost of a GC collection is likely more than manually freeing stuff, depending on the situation; but there's the tradeoff of spending easily 3x to 4x more effort in writing correct pointer code under manual MM and *still* running the risk of pointer bugs (with the associated debugging effort required to fix them -- and pointer bugs are notorious for being hard to debug), vs. paying for a (usually) small performance cost and being much more productive, with the benefit of eliminating a whole class of pointer bugs "for free".


T

-- 
Study gravitation, it's a field with a lot of potential.

On Sun, 2019-01-06 at 13:28 -0800, Walter Bright via Digitalmars-d wrote:
> 
[…]
> 1. Java has a very constrained interface to C, with a lot of rules to
> follow,
> mainly so the GC is not corrupted. D, being a systems programming language,
> simply does not have that luxury.

I understand that the Java language and its heap are very different from D, so the context and constraints are different. I am assuming Go is the same, though a priori I would think less different to D than Java. My concern (if it can be called that) is that the JVM folk and the Go folk spend a lot of time finding new and/or improved GC algorithms, whereas in D there appears to be no movement from an old and simple GC.

> 2. Let me know when Java lets go of write barriers!

G1 GC certainly avoids write barriers whenever it can. I do not know the details, but there are conference papers out there looking at avoiding write barriers at all costs in the newer GC algorithms.

-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk