On Friday, 15 October 2021 at 16:44:57 UTC, deadalnix wrote:

Is there anything we can do to start addressing this, and the other woes people have brought up?

The answer is likely no, so I foresee these same things popping up over and over and over again with no resolution since we seem to have dug ourselves into a corner.

On Friday, 15 October 2021 at 21:55:55 UTC, SealabJaster wrote:

Well at some point we need to make choices about the invariant we want and stick with them.

Considering D's values, I'd say the solution that make sense is to allocate using the GC, have an ownership system on top of it that can free most things safely. You'll note that is is possible to build a GC that is as fast as malloc free (for instance, I was able to build a GC on top of jemalloc, and the only thing I had to add is one hashmap store when allocating chunk of size bigger than 4MB. this happens almost never and when it does, memory allocation is usually not the problem as simply filling these 4MB with literally anything is far more work than the work the allocator does (to be 100% fair, this doesn't work with all mallocs, a tcmalloc style structure wouldn't cut it for instance, but jemalloc is notorious to be one of the fastest and bestest so we are not making a big concession).

So using a GC instead of malloc/free is very much viable. In fact, because we know if something is shared or not, I bet we can do something that is even faster than traditional mallocs.

So we allocate on the GC, always and use ownership on top of it. Ownership should allow you to express common allocation/disposition patterns, and when you can't, either you leak to the GC, or you free explicitly, which is unsafe.

That require repurposing @nogc as preventing leaking from the ownership system to the GC rather than allocating on the GC - which in itself is no problem if the allocations are freed properly.

There are a ton of added benefit that come attached to this if done properly, like solving a ton of problems with shared.

Even tricky situations such as @nogc exception become trivial in that model. Consider:

void foo() {
    // Allocate the exception using the GC. The exception is
    // an owned(Exception), and the corresponding overload from
    // the runtime is called.
    // Ownership of the exception is transferred to the runtime.
    // The ownership system didn't lose track of the exception, so
    // foo can be @nogc .
    throw new Exception();
}

void bar() {
    try {
        foo();
    } catch(Exception e) {
        // The ownership of the Exception is transferred back to the
        // user's code. The Exception doesn't leave the catch block,
        // so it can be freed automatically without the user having to
        // do anything.
        // In practice, that require the runtime to provide the
        // exception handler with knowledge of the exception being
        // owned or not and a runtime check, so that, if it is owned
        // it is freed.
        // bar can also be @nogc , because either the exception is owned
        // in which case it is freed and no leak occurred, or the exception
        // is not owned, in which case the leak happened somewhere else.
        // In that case, this somewhere else cannot be @nogc .
    }
}

You'll note that the ownership system does not need to be as rich as Rust's, because there is always the option to fallback on the GC or on unsafe constructs.

On Friday, 15 October 2021 at 17:17:33 UTC, Araq wrote:

I was thinking. One way for D which has common GC/raw pointers is to always allocate metadata at negative offsets. Then we D compiler can have hooks that runs when a pointer is copied, moved or goes out of scope.

At negative offsets can have a flag field that tells us if the memory is GC allocated or not. If it is not GC it just jumps over the hook code.

Drawbacks with this solution is the the raw C malloc/free cannot be used and a wrapped D version must be used that allocates additional space for the metadata. Also non GC memory will have a penalty with the hooks, as it needs to check the type all the time. For the current GC these hooks would be empty though.

Is this penalty we can live with?

On Friday, 15 October 2021 at 23:58:34 UTC, deadalnix wrote:

void foo() {
    // Allocate the exception using the GC. The exception is
    // an owned(Exception), and the corresponding overload from
    // the runtime is called.
    // Ownership of the exception is transferred to the runtime.
    // The ownership system didn't lose track of the exception, so
    // foo can be @nogc .
    throw new Exception();
}

void bar() {
    try {
        foo();
    } catch(Exception e) {
        // The ownership of the Exception is transferred back to the
        // user's code. The Exception doesn't leave the catch block,
        // so it can be freed automatically without the user having to
        // do anything.
        // In practice, that require the runtime to provide the
        // exception handler with knowledge of the exception being
        // owned or not and a runtime check, so that, if it is owned
        // it is freed.
        // bar can also be @nogc , because either the exception is owned
        // in which case it is freed and no leak occurred, or the exception
        // is not owned, in which case the leak happened somewhere else.
        // In that case, this somewhere else cannot be @nogc .
    }
}

What is your plan to make GC incremental? what about latency sensitive applications?

Can you scale the GC? servers with heaps above 1TB?

Both Java and Go can scale their GC and ensure sub 1ms collection without stopping the world

I hear you want to stick to the GC no matter what, did you know to target 60 FPS in games, one only has a budget of 16ms per frame

Languages with GC can afford one because their GC is competitive and they put lot of R&D in constantly improving and tuning them

I'm not saying GC is bad, i'm saying if you want someone to take you seriously, you have to provide serious tools so you can assist them whenever they need to scale their buisnesses

Look at ASP.NET team at microsoft, they all working towards improving the C#language so it is much less reliant on the GC

Span, value task, ref, stack alloc and bunch of other stuff they have planned for C# 10

D's history is C/C++, why stray away from that history?

On Saturday, 16 October 2021 at 00:16:07 UTC, IGotD- wrote:

Probably but it's a hack, the proper solution is to fix the type system. And since this the topic already: Not distinguishing between traced and untraced pointers (or between owned and unowned pointers for that matter) is indeed one of the "worst" language features.

On Friday, 15 October 2021 at 21:55:55 UTC, SealabJaster wrote:

In the end, i think it doesn't really matter to be honest, as long as we make sure that additions to the STD doesn't make assumption about the memory allocation strategy, we'll be fine

The library authors will be free to use what ever is best for their libraries/program

STD will be here to assist them, rather than enforce them into something

One thing to mention is that entertainment is, more and more, and faster than ever, being consumed online, in dematerialized ways, that includes games and anything VR/AR/XR

There will be a massive need of software being built, games being created for various kind devices and for multiple experiences

WASM will be important, game engines/framework built for it will be needed

And those kind of libraries will avoid GC'd langauges to maximize perf

Web stuff is a problem already solved, trying to chase it is a dead end

Our best bet i believe, is believe is to target those devs, for their engine needs, they need something that helps them be in control over the memory allocations

Let them use C#, Javascript, Lua etc as scripting language if they wish, but let them consume our libraries, they'll depend on D forever, they'll appreciate D, and maybe they'll also use D for their gameplay code instead of having to glue one

Making the STD empower that is i believe important

I think the best short term move that will help in the very long term is to finally finish STD.EXPERIMENTAL.ALLOCATORS and put it in STD.MEM

It's a little bit bloated, it'll need a little lifting

What its author has to say?

On Monday, 11 October 2021 at 15:59:10 UTC, Atila Neves wrote:

Difficult to say, features may not be bad in isolation, but turn bad when combined with other features. Features can turn bad when they can be abused for more terse source code in ways the language designers did not intend, or when scripting languages are used for larger programs.

But by-name parameter passing is pretty bad. Having expressions passed into a function executed multiple times backfires real fast in imperative languages. Such bugs can be difficult to identify.

Lack of consistency (unnecessary complexity?) in the meta-programming feature set / type system, but cannot be fixed as it would become a different language.

Simplify the language.

Flow typing.

A type system that makes it possible for the compiler to reason about ownership.

Implementation: clean separation between front-end and back-end.

On Monday, 11 October 2021 at 15:59:10 UTC, Atila Neves wrote:

• records implementation
• object promotion
• array covariance
• half baked generics
• operator overloading verbosity
• mutability by default
• New nullable semantics

• Module private instead of class private
• Integer promotion
• Phobos names (cryptic, completely different from other languages)
• Autodecoding
• Foreach_reverse (useless)
• Two mechanism for foreach (opApply & popFront)
• Traits as a library
• Real (D standardised primitive types, but here it comes with a machine dependent one)
• ff as init value for chars
• Dub as package manager
• Unqualified parameter passing (if a parameter is ref, you must pass it prefixed with ref)

• Unittest naming
• Benchmarks included in language similar to unittests
• Templated class/interface inheritance
• Shared (truly implemented)
• Cent (truly implemented)
• Range concept as language built in
• Safe by default
• Nullable/Optional as language feature

On Saturday, 16 October 2021 at 04:03:29 UTC, russhy wrote:

I asserted to write an elaborated answered and then decided to scrap it. Your answer misses the point to a degree that qualifying it of an answer is a stretch to begin with.

Long story short: none of the shit is needed if one can free explicitely, and collection cycle only are required when you leak, so really not a problem for anyone who don't leak.

You'll get the same performance profile as malloc/free. End of story.

Because we already have C and C++.