On Tuesday, 9 February 2016 at 13:01:29 UTC, NX wrote:
> On Monday, 8 February 2016 at 22:21:50 UTC, Laeeth Isharc wrote:
>> The GC itself may still be far from perfect but its much better than it was, and there are more options now.  I have found emsi containers (built on top of Andrei's allocator) pretty nice myself for my own use.
>
> Well, GC being better than it used to be doesn't change the fact it's still the worst of it's kind. I don't know if this[1] work actually got released or merged but looks like it's abandoned. Pretty sad as it seemed very promising.
>
> Anyway, I was expecting a lot more people to tell their specific problems, like "bla bla design desicion makes ARC incredibly dangerous and we can't properly interface with Objective-C without that" or like "bla bla D feature overlaps with some other stuff and requires redesign to be solved" or maybe "being unsafe (@system) by default breaks the deal"...
> GC is just one of the hundreds of problems with D and it was an example rather than the main point in this thread but thanks for anyone who replied.
>
>
> [1] http://forum.dlang.org/thread/mailman.655.1399956110.2907.digitalmars-d@puremagic.com

Thanks for pointing this one out. Opportunity comes dressed in work clothes,and I guess that until someone takes the initiative to integrate this with the newest version of the runtime / GC then nothing will happen. It's not true that there are no professional opportunities in D,  as some people say, and I can say that for some people at least impressive contributions to the language and community have paid off personally even though it was a labour of love and not motivated by that.  Good programmers don't grow on trees, and one benefit of the current size of the D community is that it's easier to make an impact and easier to stand out than in a much more crowded and mature domain where one person can only hope to achieve incremental progress.

My impression is that barriers to adoption are fairly well understood by now and it's a matter of time and hard work for them to be addressed step by step. It's not only addressing negatives but also completing positive things that will help.   Ndslice and porting BLAS on the numerical side and the interface with R will both increase the attractiveness of D on finance,  not a small area.   It's not yet mature,  but knowing one can use all the R libraries is already a big win.

On Tuesday, 9 February 2016 at 13:41:30 UTC, NX wrote:
> There are several reasons I want to use D rather than C# / Go / something else:
> - Interfacing with native API without jumping through hoops

Well, but the hoops are there to get safe and fast GC.


> - Incredibly high abstraction and meta-programming possibilities with relatively easier syntax + semantics.

Not incredibly high level abstraction... But I get what you mean. It is fairly high level for a low level language.


> - Having GC (but not a horribly slow one)

So you want this to be worked on (as D has a horribly slow one)?


> - Not bound to a specific platform (unlike C#, easier to do cross-platform work in many cases)

Doesn't C# work just as well as D (or better) with most platforms?

On Tuesday, 9 February 2016 at 13:01:29 UTC, NX wrote:
> Well, GC being better than it used to be doesn't change the fact it's still the > worst of it's kind. I don't know if this[1] work actually got released or merged but looks like it's abandoned. Pretty sad as it seemed very promising.
> [1] http://forum.dlang.org/thread/mailman.655.1399956110.2907.digitalmars-d@puremagic.com

It looks like interesting stuff, but the guy last posted in 2014. In other posts, people asked him for the code and I don't see anything on the forum indicating that he provided it. Probably an important step to improving the GC...

DMD 2.067 had some garbage collector improvements, but I'm not sure how influenced those would have been by this.

On Tue, 09 Feb 2016 14:35:48 +0000, Ola Fosheim Grøstad wrote:

> On Tuesday, 9 February 2016 at 13:41:30 UTC, NX wrote:
>> There are several reasons I want to use D rather than C# / Go /
>> something else:
>> - Interfacing with native API without jumping through hoops
> 
> Well, but the hoops are there to get safe and fast GC.
> 
> 
>> - Incredibly high abstraction and meta-programming possibilities with relatively easier syntax + semantics.
> 
> Not incredibly high level abstraction... But I get what you mean. It is fairly high level for a low level language.
> 
> 
>> - Having GC (but not a horribly slow one)
> 
> So you want this to be worked on (as D has a horribly slow one)?
> 
> 
>> - Not bound to a specific platform (unlike C#, easier to do
>> cross-platform work in many cases)
> 
> Doesn't C# work just as well as D (or better) with most platforms?

If you develop against .NET on Windows, you have a moderate chance of producing something non-portable. If you develop against Mono on Linux, you can produce something more portable more easily.

Mono, by the way, has a good garbage collector, and .NET probably has better. Mono advertises:

* precise scanning for stack, heap, and registers
* generational collection using write barriers
* per-thread sub-heaps for faster allocation
* multithreaded scanning

I think D could implement all that with a couple caveats.

Write barriers are a problem for real-time code. Right now, we can tell you: you can write code with real-time sections as long as you don't allocate GC memory in the real-time sections.

If we introduced write barriers, well, the most straightforward way of doing that is to use memory protection and install a fault handler. If you write to a page that hasn't been written to since the last collection, you get a page fault, the kernel dispatches it to your fault handler, the fault handler marks a "card" (it sets a boolean corresponding to the page you tried to write to). Then the handler marks that page as writable and you go on with your day.

Alternatively, the compiler could insert code at every pointer write. (This is the deamortized version. Consistent latency, but if you write to the same pointer variable a million times between GC allocations, you pay that cost a million times more than you really need to.) You would need a compiler switch to disable this behavior. That's not likely to happen.

Walter and Andrei would not accept that, since it makes it difficult to get bare-metal performance (and also makes it harder to interface with C).

Which leads me to another thing holding D back. What use cases are we trying to support? What are we trying to optimize for? Apparently everything. That doesn't work. To some extent you can make things faster in general, but you'll still end up supporting all use cases moderately well and none exceedingly well.

Anyway, another GC issue is supporting generational collection. You need a moving GC to make it work efficiently. D could support a moving collector, but it would require more runtime type information. (And stack maps, but that might be there already. Not sure.) Walter has been strongly against adding extra runtime type information in the past.

On Tuesday, 9 February 2016 at 14:35:48 UTC, Ola Fosheim Grøstad wrote:
> Not incredibly high level abstraction... But I get what you mean. It is fairly high level for a low level language.

Coming from C#, it looks amazing but probably not that incredible when coming from C++.


> So you want this to be worked on (as D has a horribly slow one)?

I would want it to be solved rather than being worked on... which requires design change which is probably not going to happen. There is still room for improvement though.


> Doesn't C# work just as well as D (or better) with most platforms?

There are differences, but yeah I shouldn't have said that ~ cross-platform thingy is not a valid argument against C# anymore.

On Tuesday, 9 February 2016 at 17:41:34 UTC, NX wrote:

> I would want it to be solved rather than being worked on... which requires design change which is probably not going to happen. There is still room for improvement though.

Right. I think there are at least two things that can improve current GC without any changes in design: parallel marking and lazy sweeping. Currently (at least last time I checked) GC pauses the world, then does all the marking in one thread, then all the sweeping. We can do the marking in several parallel threads (this is much harder to implement but still doable), and we can kick the sweeping out of stop-the-world pause and do the sweeping lazily: when you try to allocate some memory it will not just look in free lists, it will try to collect some unused unswept memory from the heap first. This way allocations become a bit slower but GC pause time reduces significantly. Concurrent sweeping is another possibility.
Of course, it's all easier said than done, without an actual hero who would code this, it remains just talk.

On Tuesday, 9 February 2016 at 17:41:34 UTC, NX wrote:
> On Tuesday, 9 February 2016 at 14:35:48 UTC, Ola Fosheim Grøstad wrote:
>> Not incredibly high level abstraction... But I get what you mean. It is fairly high level for a low level language.
>
> Coming from C#, it looks amazing but probably not that incredible when coming from C++.

If D had parity with C++ features and didn't have it's own set of inconsistencies it could do a lot better. But the lack of willingness to change probably means that a new language will come along instead of D improving.

Right now D sits somewhere between C++/Rust and Swift/Go, which might not be the best position. It is natural that people will gravitate towards either end. If you don't want to deal with memory management use Swift/Go, if you want to optimize memory management go C++/Rust.


>> So you want this to be worked on (as D has a horribly slow one)?
>
> I would want it to be solved rather than being worked on... which requires design change which is probably not going to happen. There is still room for improvement though.

I think it would help a lot if there was a broad effort to refactor and document the compiler. If the compiler is difficult to change it becomes more difficult to experiment and there will be much more resistance to changing semantics...

On Wed, 10 Feb 2016 08:57:51 +0000, thedeemon wrote:

> Currently (at least last time I checked) GC pauses the world, then does all the marking in one thread, then all the sweeping.

Right.

> We can do the
> marking in several parallel threads (this is much harder to implement
> but still doable),

Parallel marking would not be a breaking change by any means. No user code runs during GC collections, so we can do anything. The major fly in the ointment is that creating threads normally invokes the GC, since Thread is an object, and invoking the GC during a collection isn't the best.

This can be solved by preallocating several mark threads. Then you just divide the stack and roots between those threads. Moderately annoying sync issues

This doesn't guarantee an even distribution of work. You can solve that problem with a queue, though that requires locking.

The main wrinkle is writing a bit to shared data structures, which can be slow. On the other hand, in the mark phase, we're only ever going to write the same value to each, so it doesn't matter if GC thread A . I don't know how to tell the CPU that it doesn't have to read back the memory before writing it.

> and we can kick the sweeping out of stop-the-world
> pause and do the sweeping lazily

This would be a breaking change. Right now, your destructors are guaranteed to run when no other code is running. You'd need to introduce locks in a few places.

I'm not saying this is a bad thing. I think people generally wouldn't notice if we made this change. But some code would break, so we'd have to stage that change.

Anyway, I'm hacking up parallel mark phase to see how it would work. I could use some GC benchmarks if anyone's got them lying around.

On Wednesday, 10 February 2016 at 20:21:22 UTC, Chris Wright wrote:
> On Wed, 10 Feb 2016 08:57:51 +0000, thedeemon wrote:
>
>> Currently (at least last time I checked) GC pauses the world, then does all the marking in one thread, then all the sweeping.
>
> Right.
>
>> We can do the
>> marking in several parallel threads (this is much harder to implement
>> but still doable),
>
> Parallel marking would not be a breaking change by any means. No user code runs during GC collections, so we can do anything. The major fly in the ointment is that creating threads normally invokes the GC, since Thread is an object, and invoking the GC during a collection isn't the best.
>
> This can be solved by preallocating several mark threads. Then you just divide the stack and roots between those threads. Moderately annoying sync issues
>
> This doesn't guarantee an even distribution of work. You can solve that problem with a queue, though that requires locking.
>
> The main wrinkle is writing a bit to shared data structures, which can be slow. On the other hand, in the mark phase, we're only ever going to write the same value to each, so it doesn't matter if GC thread A . I don't know how to tell the CPU that it doesn't have to read back the memory before writing it.
>
>> and we can kick the sweeping out of stop-the-world
>> pause and do the sweeping lazily
>
> This would be a breaking change. Right now, your destructors are guaranteed to run when no other code is running. You'd need to introduce locks in a few places.
>
> I'm not saying this is a bad thing. I think people generally wouldn't notice if we made this change. But some code would break, so we'd have to stage that change.
>
> Anyway, I'm hacking up parallel mark phase to see how it would work. I could use some GC benchmarks if anyone's got them lying around.

https://github.com/D-Programming-Language/druntime/tree/master/benchmark/gcbench

On Tuesday, 9 February 2016 at 13:41:30 UTC, NX wrote:
> There are several reasons I want to use D rather than C# / Go / something else:

I will focus on comparing against C++, because that has been my favorite general purpose language for a long time. While I often have to use C, Java, C#, etc. for various business reasons, when faced with the choice on pure technical merits I will go for C++ any day (haven't tried Go, but was unimpressed by my initial read-over). D is the first language I have ever encountered with a serious chance of unseating C++ as my personal favorite.

> - Interfacing with native API without jumping through hoops

Concur. Though I get this with C++, too.

> - Incredibly high abstraction and meta-programming possibilities with relatively easier syntax + semantics.

Yes. The lack of powerful meta-programming is so frustrating in languages like Java and C#. C++ and D both have the power, but only D has the ease of reading and writing.

> - It's harder to reverse engineer native code than byte code equivalent.

Meh. True, but this doesn't do much for me; it still isn't -that- hard to reverse native code, at least to the point of exploitation (to the point of copying is much harder). It just takes longer.

> - Trading off anything according to your needs.

Yes. This is critical. I actually feel like D does this a little worse than C++ (though not significantly so), if only because it is difficult to completely avoid the GC, and if you want to avoid it and still use inheritance you need to break out the custom allocators. Most of the time this isn't a problem.

> - Expressiveness and purity, immutablity concepts.

Expressiveness is key, though I haven't found D to be terribly more expressive than C++. A little better here, a little worse there. On the other hand, it is usually syntactically nicer when expressing concepts, sometimes greatly so.

Immutability is nice. The attention paid to threading was what caused me to take a closer look at D in the first place.

> - Having GC (but not a horribly slow one)

Meh. I know there are things which are much easier to express with a GC, but they don't really come up for me. On the other hand, I often need deterministic cleanup, so the GC can be kind of an annoyance, since it lends itself to a lot of wrapping things in structs and forcing me to pay more attention to lifetime rules than I have to in C++.

The other (main?) purported benefits of a GC (avoiding leaks and dangling pointers) don't do much for me, since it is almost trivially easy to avoid those problems in C++ anyway, without introducing the headaches of the GC; certainly it is easier than the focus I have to give D object lifetimes now. That may be a matter of relative practice, though, since I've used C++ for a long long time and D for...3 weeks? :)

> - Syntactic sugars (associtive arrays, powerful foreach, slices...)

I'm still adjusting to the idea of AAs as part of the language rather than library. Not sure I like it, but on the other hand it doesn't really hurt. The foreach construct isn't any better (or worse) than C++'s, unless I'm missing something (which is very possible). But slices are awesome!

> - Compile times

Oh god yes. This makes metaprogramming so much more palatable.

> - Not bound to a specific platform (unlike C#, easier to do cross-platform work in many cases)

I'll go it one step further, and note that D feels more portable than C++ to me...at least to the major platforms I usually work on. Maybe it's the simple fact that things like sockets are defined in the libraries, or that I don't have to #include <Windows.h> :).

> I wish D could be better. I really want it with all of my heart...

D has a lot to offer. Here are a few other things I've really liked over C++:
* Modules. C++ is supposed(?) to get them at some point I suppose, but for here and now it's a clear advantage for D.
* Not syntactically separating interface and implementation (e.g., C++'s header vs source file dichotomy). This was never a true separation in C++, and just led to lots of extra syntax and minor DRY violations. Of course you could write everything inline anyway...until it depended on something declared later.
* Related to the point above, not having to think about whether to make something inline. Sure, C++ compilers make that choice for you, but you still have to decide whether to allow them (or at least the ones without link-time code generation) by putting your source in the header file. Needless headache for something a compiler can do.
* Properly doing away with the C preprocessor. I haven't seen a need for it that wasn't addressed by another D feature.
* Properly doing away with MI. Unlike some languages which just canned it, D actually replaced its functionality with other features.
* Thread-local by default. So simple. So useful.
* The in keyword. This is nice syntactic sugar over having a special trait in C++ which deduces whether to pass by value or const-reference. "foo(in bar)" is way more readable than something like "foo(traits<bar>::fast_param bar)"
* Convenient operator overloading for numerical types.
* @property. This little feature has been invaluable in porting my C++ code, letting me shave off tons of accessors and mutators that existed only for the sake of possibly being needed in the future. I didn't even need to use @property for this; its simple existence did the work for me!

There are others. Lots of others. Basically they boil down to: D mostly feels like C++, but just easier to read and write. Which is fantastic!

Of course, it's not all rosy. There has been some debate on the General forum about D's ability to express move semantics. I'm wary of the GC. Some language behaviors seem unintuitive (which can lead to bugs), like how you can store keys in AA that are null. And probably the biggest thing I don't like about D isn't the language design, but the immaturity of DMD. I've only used D for a few weeks, and I've already reported three bugs against it. This means that unlike with most languages, I don't have confidence that program failures are really my fault as opposed to just another compiler bug.

Summary: D has a lot of potential. With a little maturation, it could be a serious rival to C++.

Oh, and for whatever it's worth, I'm all for breaking changes if they improve the language. C++ has long suffered from unwillingness to do this. With a proper deprecation path, it can be a palatable way to continue evolving the language and avoiding stagnation.