On Sunday, 6 December 2020 at 10:44:39 UTC, Max Haughton wrote:
> On Sunday, 6 December 2020 at 05:29:37 UTC, Ola Fosheim Grostad wrote:
> It has to be either some kind of heavily customisable small GC (i.e. with our resources the GC cannot please everyone), or arc. The GC as it is just hurts the language.
>
> Realistically, we probably need some kind of working group or at least serious discussion to really narrow down where to go in the future. The GC as it is now must go, we need borrowing to work with more than just pointers, etc.
>
> The issue is that it can't just be done incrementally, it needs to be specified beforehand.

ARC can be done incrementally, we can do it as a library first and use a modified version existing GC for detecting failed borrows at runtime during testing.

But all libraries that use owning pointers need ownership to be made explicit.

A static borrow checker an ARC optimizer needs a high level IR though. A lot of work though.

On Sunday, 6 December 2020 at 11:07:50 UTC, Ola Fosheim Grostad wrote:
> On Sunday, 6 December 2020 at 10:44:39 UTC, Max Haughton wrote:
>> On Sunday, 6 December 2020 at 05:29:37 UTC, Ola Fosheim Grostad wrote:
>> It has to be either some kind of heavily customisable small GC (i.e. with our resources the GC cannot please everyone), or arc. The GC as it is just hurts the language.
>>
>> Realistically, we probably need some kind of working group or at least serious discussion to really narrow down where to go in the future. The GC as it is now must go, we need borrowing to work with more than just pointers, etc.
>>
>> The issue is that it can't just be done incrementally, it needs to be specified beforehand.
>
> ARC can be done incrementally, we can do it as a library first and use a modified version existing GC for detecting failed borrows at runtime during testing.
>
> But all libraries that use owning pointers need ownership to be made explicit.
>
> A static borrow checker an ARC optimizer needs a high level IR though. A lot of work though.

ARC with a library will have overhead unless the compiler/ABI is changed e.g. unique_ptr in C++ has an indirection.

The AST effectively is a high-level IR. Not a good one, but good enough. The system Walter has built shows the means are there in the compiler already.

As things are at the moment, the annotations we have for pointers like scope go a long way, but the language doesn't deal with things like borrowing structs (and the contents of structs i.e. making a safe vector) properly yet. That is what needs thinking about.

On Sunday, 6 December 2020 at 11:27:39 UTC, Max Haughton wrote:
> ARC with a library will have overhead unless the compiler/ABI is changed e.g. unique_ptr in C++ has an indirection.

No, unique doesnt need indirection, neither does ARC, we put the ref count at a negative offset.

shared_ptr is a fat pointer with the ref count as a separate object to support existing C libraries, and make weak_ptr easy to implement. But no need for indirection.

> The AST effectively is a high-level IR. Not a good one, but good enough. The system Walter has built shows the means are there in the compiler already.

I think you need a new IR, but it does not have to be used for code gen, it can point back to the ast nodes that represent ARC pointer assignments.

One could probably translate the one used in Rust, even.

On Sunday, 6 December 2020 at 11:07:50 UTC, Ola Fosheim Grostad wrote:
>
> ARC can be done incrementally, we can do it as a library first and use a modified version existing GC for detecting failed borrows at runtime during testing.
>
> But all libraries that use owning pointers need ownership to be made explicit.
>
> A static borrow checker an ARC optimizer needs a high level IR though. A lot of work though.

The Rust approach is interesting as it doesn't need an ARC optimizer. Everything is a  move so no increase/decrease is done when doing that. Increase is done first when the programmer decides to 'clone' the reference. This inherently becomes optimized without any compiler support. However, this requires that the programmer inserts 'clone' when necessary so it isn't really automatic.

I was thinking about how to deal with this in D and the question is if it would be better to be able to control move as default per type basis. This way we can implement Rust style reference counting without intruding too much on the rest of the language. The question is if we want this or if we should go for a fully automated approach where the programmer doesn't need to worry about 'clone'.

On Sunday, 6 December 2020 at 12:58:44 UTC, IGotD- wrote:
> I was thinking about how to deal with this in D and the question is if it would be better to be able to control move as default per type basis. This way we can implement Rust style reference counting without intruding too much on the rest of the language. The question is if we want this or if we should go for a fully automated approach where the programmer doesn't need to worry about 'clone'.

I dont know, but I suspect that people that use D want something more high level than Rust? But I dont use Rust, so...

On Sunday, 6 December 2020 at 11:35:17 UTC, Ola Fosheim Grostad wrote:
> On Sunday, 6 December 2020 at 11:27:39 UTC, Max Haughton wrote:
>> [...]
>
> No, unique doesnt need indirection, neither does ARC, we put the ref count at a negative offset.
>
> shared_ptr is a fat pointer with the ref count as a separate object to support existing C libraries, and make weak_ptr easy to implement. But no need for indirection.
>
>> [...]
>
> I think you need a new IR, but it does not have to be used for code gen, it can point back to the ast nodes that represent ARC pointer assignments.
>
> One could probably translate the one used in Rust, even.

https://gcc.godbolt.org/z/bnbMeY

On Sunday, 6 December 2020 at 08:12:58 UTC, Ola Fosheim Grostad wrote:
> On Sunday, 6 December 2020 at 07:45:17 UTC, Bruce Carneal wrote:
>> GCs scan memory, sure.  Lots of variations.  Not germane.  Not a rationale.
>
> We need to freeze the threads when collecting stacks/globals.
>
>> D is employed at multiple "levels".  Whatever level you call it, Go and modern JVMs employ low latency GCs in multi-threaded environments.  Some people would like to use D at that "level".
>
> Yes, but they don't allow low level programming. Go also freeze to sync threads this has a rather profound impact on code generation. They have spent a lot of effort on  sync instructions in code gen to lower the latency AFAIK.
>

They surely do.

Looking forward to see D achieve the same performance level as .NET 5 is capable of, beating Google's own gRPC C++ implementation, only Rust implementation beats it.

https://www.infoq.com/news/2020/12/aspnet-core-improvement-dotnet-5/

And while on the subject of low level programming in JVM or .NET.

https://www.infoq.com/news/2020/12/net-5-runtime-improvements/

> Many of the performance improvements in the HTTP/2 implementation are related to the reimplementation from unmanaged C++ code to managed C# code. Lander notes that there "still is this kind of idea that managed languages are not quite up to the task for some of those low-level super performance sensitive components,

Rich Lander being one of the main .NET architects, and upcoming Java 16 features, http://openjdk.java.net/jeps/389 (JNI replacement), http://openjdk.java.net/jeps/393 (native memory management).

As I already mentioned in another thread, rebooting the language to pull in imaginary crowds will only do more damage than good, while the ones deemed unusable by the same imaginary crowd just keep winning market share, slowly and steady, even if takes yet another couple of years.

On Sunday, 6 December 2020 at 08:59:49 UTC, Ola Fosheim Grostad wrote:
> On Sunday, 6 December 2020 at 08:36:49 UTC, Bruce Carneal wrote:
>>> Yes, but they don't allow low level programming. Go also freeze to sync threads this has a rather profound impact on code generation. They have spent a lot of effort on  sync instructions in code gen to lower the latency AFAIK.
>>
>> So, much of the difficulty in bringing low-latency GC to dlang would be the large code gen changes required.  If it is a really big effort then that is all we need to know.  Not worth it until we can see a big payoff and have more resources.
>
> Well, you could in theory avoid putting owning pointers on the stack/globals or require that they are registered as gc roots. Then you don't have to scan the stack. All you need then is write barriers. IIRC

'shared' with teeth?

On Sunday, 6 December 2020 at 14:45:21 UTC, Bruce Carneal wrote:
>> Well, you could in theory avoid putting owning pointers on the stack/globals or require that they are registered as gc roots. Then you don't have to scan the stack. All you need then is write barriers. IIRC
>
> 'shared' with teeth?

It was more a hypothetical, as read barriers are too expensive. But write barriers should be ok, so a single-threaded incremental collector could work well if D takes a principled stance on objects not being 'shared' not being handed over to other threads without pinning them in the GC.

Maybe a better option for D than ARC, as it is closer to what people are used to.

On Sunday, 6 December 2020 at 14:11:41 UTC, Max Haughton wrote:
> On Sunday, 6 December 2020 at 11:35:17 UTC, Ola Fosheim Grostad wrote:
>> On Sunday, 6 December 2020 at 11:27:39 UTC, Max Haughton wrote:
>>> [...]
>>
>> No, unique doesnt need indirection, neither does ARC, we put the ref count at a negative offset.
>>
>> shared_ptr is a fat pointer with the ref count as a separate object to support existing C libraries, and make weak_ptr easy to implement. But no need for indirection.
>>
>>> [...]
>>
>> I think you need a new IR, but it does not have to be used for code gen, it can point back to the ast nodes that represent ARC pointer assignments.
>>
>> One could probably translate the one used in Rust, even.
>
> https://gcc.godbolt.org/z/bnbMeY

If you pass something as a parameter then there may or may not be an extra reference involved. Not specific for smart pointers, but ARC optimization should take care of that.