October 25, 2021

On Monday, 25 October 2021 at 18:04:19 UTC, IGotD- wrote:

>

Then we have the shared structs/classes in D where all basic types are forced to be atomic, which is totally insane.

As far as I know this is not what shared does, and not what shared is intended to do. By itself, shared is just a marker for data that requires synchronization to access (what the spec calls "shared memory locations"). Whether that synchronization is accomplished using atomic operations or locking is entirely up to the programmer.

October 25, 2021

On Monday, 25 October 2021 at 18:12:36 UTC, Paul Backus wrote:

>

As far as I know this is not what shared does, and not what shared is intended to do. By itself, shared is just a marker for data that requires synchronization to access (what the spec calls "shared memory locations"). Whether that synchronization is accomplished using atomic operations or locking is entirely up to the programmer.

Last time I tried a shared struct I had to cast away the atomic operations on about every line. Is this removed in newer compiler versions?

October 25, 2021
On Monday, 25 October 2021 at 18:04:19 UTC, IGotD- wrote:
> Always take the lock, don't dwell into atomic operations/structures too much. 99% of the cases should be handled by traditional mutex/semaphores and possible language layers above that.

Hm, in real-time scenarios you cannot take locks, so for system level programming you have to deal with atomics. It does get tricky real fast though… so make the data model as simple as possible and don't focus too much on micro-optimization in the beginning (just be happy if you can convince yourself that it is provably correct).

> Synchronized classes are good because the lock is implicit. However, it might be inefficient if you use several methods after each other which means several lock/unlock (Acquire/Release or whatever you name it) after each other.

Yes, I wouldn't have added the feature, but once it is there it is an easy to understand starting point. Monitors is also something that is taught at universities so it is a concept "generic" programmers would (should) know.

> One of the best designs in Rust was to combine the borrowing with acquiring the lock. Then you can borrow/lock on a structure do all the operations you want as in normal single threaded programming and it will be released automatically when the borrow goes out of scope. This is is a genius design and I'm not sure how to pry it into D.

I don't know enough about Rust's approach here, but in general, effective locking strategies cannot easily be formulated in a way that a C-ish language compiler can reason about. For that you would need a much more advanced layer, I think. So, not really realistic for D in the general case(and probably not for Rust either).


> Then we have the shared structs/classes in D where all basic types are forced to be atomic, which is totally insane.

This is the case where D gets bitten by "being pragmatic" because of users pushing for convenience. And it isn't safe to allow access to shared state that way, or rather; it does not encourage correctness. Meaning, it would only be safe for very simple models.

What is needed is some kind of effect system that where "shared unaccessible" is turned into "shared accessible" (but not non-shared). I guess this is what Rust kinda touches upon. This might be difficult to do properly, so it can also be unverified by the compiler, but you should be able to draw "lines in the sand" in your code, somehow.

(I don't have the answer, but I think there are opportunities.)





October 25, 2021

On Monday, 25 October 2021 at 18:12:36 UTC, Paul Backus wrote:

>

As far as I know this is not what shared does, and not what shared is intended to do. By itself, shared is just a marker for data that requires synchronization to access (what [the spec][1] calls "shared memory locations"). Whether that synchronization is accomplished using atomic operations or locking is entirely up to the programmer.

I think shared is not usable as it stands today, so I have no experience with it, but unless my memory is playing tricks with me; I think someone pushed for allowing atomic access to struct fields of a shared struct and that this was accepted by Walter? If I am wrong, then I apologize.

October 25, 2021

On Monday, 25 October 2021 at 18:22:59 UTC, IGotD- wrote:

>

On Monday, 25 October 2021 at 18:12:36 UTC, Paul Backus wrote:

>

As far as I know this is not what shared does, and not what shared is intended to do. By itself, shared is just a marker for data that requires synchronization to access (what the spec calls "shared memory locations"). Whether that synchronization is accomplished using atomic operations or locking is entirely up to the programmer.

Last time I tried a shared struct I had to cast away the atomic operations on about every line. Is this removed in newer compiler versions?

I think perhaps you are mistaking the suggestion to use atomic operations in the compiler's error message for the actual presence of atomic operations in the code.

shared int x;

void main()
{
    x += 1;
    // Error: read-modify-write operations are not allowed for `shared` variables
    //        Use `core.atomic.atomicOp!"+="(x, 1)` instead
}

The compiler requires you to use some kind of synchronization to modify x. The error message (perhaps misguidedly) suggests using atomic operations, but they are not actually required--you could also use a mutex.

If you're using a mutex, you do have to cast away shared once you have locked it, since in general the compiler has no way of knowing which mutex is associated with which variable.

October 25, 2021

On Monday, 25 October 2021 at 18:59:51 UTC, Paul Backus wrote:

>

The compiler requires you to use some kind of synchronization to modify x. The error message (perhaps misguidedly) suggests using atomic operations, but they are not actually required--you could also use a mutex.

100% misguided, and the type system should not allow it. It cannot be assumed to be safe.

>

If you're using a mutex, you do have to cast away shared once you have locked it, since in general the compiler has no way of knowing which mutex is associated with which variable.

And this is where almost all utility of shared is lost. Now you can no longer assume that something that isn't marked as shared is thread local...

What is left is syntactical clutter.

October 25, 2021

On Monday, 25 October 2021 at 19:12:52 UTC, Ola Fosheim Grøstad wrote:

>

On Monday, 25 October 2021 at 18:59:51 UTC, Paul Backus wrote:

>

The compiler requires you to use some kind of synchronization to modify x. The error message (perhaps misguidedly) suggests using atomic operations, but they are not actually required--you could also use a mutex.

100% misguided, and the type system should not allow it. It cannot be assumed to be safe.

It can be assumed not to cause a data race, which means that an atomic operation on a shared variable is exactly as safe as the corresponding non-atomic operation on a thread-local variable.

> >

If you're using a mutex, you do have to cast away shared once you have locked it, since in general the compiler has no way of knowing which mutex is associated with which variable.

And this is where almost all utility of shared is lost. Now you can no longer assume that something that isn't marked as shared is thread local...

The language spec defines "thread local" as follows:

>

Thread-local memory locations are accessible from only one thread at a time.

And further clarifies that

>

A memory location can be temporarily transferred from shared to local if synchronization is used to prevent any other threads from accessing the memory location during the operation.

Of course, the compiler will not stop you from writing incorrect casts in @system code, but that's not an issue unique to shared.

October 25, 2021

On Monday, 25 October 2021 at 19:52:10 UTC, Paul Backus wrote:

>

It can be assumed not to cause a data race, which means that an atomic operation on a shared variable is exactly as safe as the corresponding non-atomic operation on a thread-local variable.

Of course it can't. Let take the simplest of the simple; a struct with a date with 3 fields_ day, month and year. Now, let us assume "2021-10-31". And then we add one day using atomics? How?

If I declare something as shared as a type, then I expect some solid means to protect it.

The current setup is too naive to be useful. It is no better than a wrapper-template. It does not need to implemented in the type system. It could have been implemented as a regular library.

This is not to say that the concept of "shared" is not useful. It is the might-as-well-have-been-a-template-wrapper-approach that is useless.

>

[The language spec][1] defines "thread local" as follows:

>

Thread-local memory locations are accessible from only one thread at a time.

That is way too weak to get the benefits that are desirable, meaning: a competitive edge over C++.

I can create a template-wrapper in C++ too. So, as is, "shared" provides no advantage as far as I can tell.

>

And further clarifies that

>

A memory location can be temporarily transferred from shared to local if synchronization is used to prevent any other threads from accessing the memory location during the operation.

Of course, the compiler will not stop you from writing incorrect casts in @system code, but that's not an issue unique to shared.

I don't see how this can be guaranteed. The compiler needs to know where the line in the sand is drawn so that is isn't limited by the potentially performance-limiting sequencing points that C++ has to deal with.

For instance: what are the lifetimes for cached computations when you don't know if another thread will obtain access to what you received as a "nonshared object"?

Also, in order to get solid GC performance the compiler needs to know whether the memory is owned by the thread or is foreign to it.

"shared" has to be more than a shell in order to enable more "power"...

October 25, 2021

On Monday, 25 October 2021 at 20:12:04 UTC, Ola Fosheim Grøstad wrote:

>

On Monday, 25 October 2021 at 19:52:10 UTC, Paul Backus wrote:

>

It can be assumed not to cause a data race, which means that an atomic operation on a shared variable is exactly as safe as the corresponding non-atomic operation on a thread-local variable.

Of course it can't. Let take the simplest of the simple; a struct with a date with 3 fields_ day, month and year. Now, let us assume "2021-10-31". And then we add one day using atomics? How?

I agree that trying to do this with atomics will not give you the right answer, but it is at least guaranteed not to cause undefined behavior (excluding contract/assertion failures). Safety does not imply correctness.

> > >

Thread-local memory locations are accessible from only one thread at a time.

That is way too weak to get the benefits that are desirable, meaning: a competitive edge over C++.

I can create a template-wrapper in C++ too. So, as is, "shared" provides no advantage as far as I can tell.

The advantages are:

  1. In C++, such a template wrapper would be opt-in. In D, shared is opt-out.
  2. @safe D code can assume that anything non-shared is thread-local, because only @system or @trusted code can cast to and from shared.

In other words, it makes code easier to reason about and concurrency bugs easier to isolate.

You are probably correct that shared is not very useful for enabling compiler optimizations relative to what is possible in C++.

October 25, 2021

On Monday, 25 October 2021 at 20:41:25 UTC, Paul Backus wrote:

>

In other words, it makes code easier to reason about and concurrency bugs easier to isolate.

That remains to be seen? There is really nothing that prevents another thread from writing to something that @safe code has access to. So not sure how this is a better situation than C++ has...

I somehow doubt that such surface semantics are enough for people to convince themselves that the hazzle of dealing with a feature is worth it (outside the most enthusiastic D programmers). Shared ends up a bit like transitive const and pure: you could, but won't, because it doesn't appear to provide any real edge. So why bother satisfying a whining compiler if you can avoid it altogether?

>

You are probably correct that shared is not very useful for enabling compiler optimizations relative to what is possible in C++.

It will be very difficult for D to grow its own niche if what distinguishes it from other languages is primarily on the surface level.

Rust is gaining ground on C++ because it is good at something that C++ cannot be good at, and that is probably also the only reason for why it is gaining ground?