On Saturday, October 12, 2019 3:15:26 AM MDT Ola Fosheim Grøstad via Digitalmars-d wrote:
> On Saturday, 12 October 2019 at 06:31:18 UTC, Jonathan M Davis
>
> wrote:
> > On Friday, October 11, 2019 10:58:29 PM MDT Walter Bright via
> >
> > Digitalmars-d wrote:
> >> On 10/2/2019 3:42 AM, Nicholas Wilson wrote:
> >> > It should be that shared memory access is disabled in all
> >> > contexts,
> >> > which must be worked around with casts,
> >> > which makes the function @system,
> >> > which must then be encapsulated with a @trusted interface in
> >> > order to
> >> > use in @safe code.
> >>
> >> Sounds right.
> >
> > This is pretty much what several of us were arguing for during the discussions about shared at dconf this year.
>
> How are you going to prove that @safe code does not retain nonshared-references after the lock has been released?
>
> How does his work with array elements/slices.

It's the same as with any @trusted code. It's up to the programmer to ensure that what the code is doing is actually @safe, and if the code isn't able to provide an API that is @safe, then it shouldn't be @trusted. Ultimately, if the programmer casts away shared while a mutex is locked, it's up to the programmer to ensure that no thread-local references to the shared data escape (though scope can help with that) so that when the lock is released, the only references left are shared.

- Jonathan M Davis

On Saturday, 12 October 2019 at 09:45:41 UTC, Jonathan M Davis wrote:
> It's the same as with any @trusted code. It's up to the programmer to ensure that what the code is doing is actually @safe, and if the code isn't able to provide an API that is @safe, then it shouldn't be @trusted.

But didn't Walter in another thread some time ago say that he would add Rust-like linear typing for memory management?

If this is the goal, shouldn't you also use linear typing for allowing more flexible @safe/@trusted APIs?

> Ultimately, if the programmer casts away shared while a mutex is locked, it's up to the programmer to ensure that no thread-local references to the shared data escape (though scope can help with that) so that when the lock is released, the only references left are shared.

How does that work with arrays. E.g. processing that has coroutine/ranges style implementations.  How can you ensure that it is impossible for the lock to be released and also making sure that all @safe single-threaded array processing code can be used?

You often want to use "shared" for large sections of arrays.

Another thing is that it seems to me that things could go wrong if you have slices and dynamic arrays. E.g.:

a. you take a slice of the last element of an unshared dynamic array and hand it to one thread.

b. you reduce the length of the dynamic array

c. you hand out an unshared version of the dynamic array to another thread which then increase the length of the dynamic array in @safe code

Now you have two @safe unshared references to the last element in two different threads? I don't have much experience with extending dynamic arrays, because I don't like the D semantics of it, but isn't this roughly how things could go?

On Saturday, 12 October 2019 at 10:22:38 UTC, Ola Fosheim Grøstad wrote:
> c. you hand out an unshared version of the dynamic array to another thread which then increase the length of the dynamic array in @safe code

I'm sorry. I mixed up Go and D dynamic array semantics... :-/

I thought they were exactly the same, but apparently D always reallocates while Go tries to avoid reallocation.  I dislike the semantics of Go too... that being said.

Still, it might be an issue for library types, even though it appears to work out for builtins (?).

On Saturday, 12 October 2019 at 09:45:41 UTC, Jonathan M Davis wrote:
> It's the same as with any @trusted code. It's up to the programmer to ensure that what the code is doing is actually @safe, and if the code isn't able to provide an API that is @safe, then it shouldn't be @trusted.

So, I am trying to figure out what that API would have to look like and the only solution I can think of right away is that you have are doing a callback back into @safe code from the lock-holding @trusted code?

So, if you have to obtain locks from multiple sources, it either won't work out, or you have to use some serious kludges, like a deep call chain:

while(tryagain) {
    trusted_code_obtains_lock_1-->
       safe_code_callback1 calls->
           trusted_code_obtains_lock2
               safe_code_callback2 processes and sets tryagain=false
}

Of course, that could lead to starvation.

So to avoid all of that multi-threaded libraries have to have 2 APIs, one simple transactional wrapper-API for @safe code and another API for @trusted code that can be used to write a new wrapper that allows locking of multiple datastructures.

I hope people try to sketch out what such libraries will look like before the semantics are decided on. Maybe an experimental branch would be appropriate.

On Saturday, October 12, 2019 4:22:38 AM MDT Ola Fosheim Grøstad via Digitalmars-d wrote:
> On Saturday, 12 October 2019 at 09:45:41 UTC, Jonathan M Davis
>
> wrote:
> > It's the same as with any @trusted code. It's up to the programmer to ensure that what the code is doing is actually @safe, and if the code isn't able to provide an API that is @safe, then it shouldn't be @trusted.
>
> But didn't Walter in another thread some time ago say that he would add Rust-like linear typing for memory management?
>
> If this is the goal, shouldn't you also use linear typing for allowing more flexible @safe/@trusted APIs?

I don't know anything about what you mean by linear types (I really don't know much about Rust). So, I can't say how that would come into play, but in general, with @trusted, the programmer must verify that what the code is doing is @safe and must provide an API that is @safe. If the programmer can't guarantee that what the code is doing is @safe, then they shouldn't be marking it is @trusted. Any operations that the compiler can guarantee as @safe are already considered @safe. By definition, once @system is involved, it's because the compiler can't guarantee that the code is @safe, and @trusted is the programmer's tool for telling the compiler that a particular piece of code is actually @safe even though the compiler couldn't verify that.

In the case of shared, in general, it's not thread-safe to read or write to such a variable without either using atomics or some other form of thread synchronization that is currently beyond the ability of the compiler to make guarantees about and will likely always be beyond the ability of the compiler to make guarantees about except maybe in fairly restricted circumstances. The type system separates thread-local from shared, but it doesn't have any concept of ownership and simply doesn't have the kind of information required to figure out whether something is thread-safe. The closest that anyone has proposed is TDPL's synchronized classes, and even those could only safely remove the outer layer of shared - and even that required locking down synchronized classes pretty thoroughly.

That's why it doesn't make sense to allow reading from or writing to shared variables, and it has to be up to the programmer to deal with the synchronization primitives and temporarily casting to thread-local to operate on the shared data while it's protected by those synchronization primitives. The casts automatically make such code @system as it should be, and then the programmer is going to have to guarantee the thread-safety just like they'd do in a language like Java or C++. The difference is that D's type system helps the programmer, whereas languages like Java or C++ leave it entirely up to the programmer to get it right. The type system segregates the data that's shared, and the code that does all of the tricky threading stuff is @system, requiring the programmer to verify that piece of code and mark it as @trusted to then be used by the @safe portions of the program. Yes, the casts make the code more verbose than it would be in C++, but other than that, you're basically doing exactly the same thing that you'd be doing in C++.

By no means does that mean that any of the threading stuff is easy, but as long as the type system cannot make guarantees about thread-safety, segregating the code that deals with the threading and requiring that the programmer verify its correctness in order to have it work with @safe code is the best that we can do. Regardless, even if we're able to later come up with improvements that allow shared to be implicitly removed under some set of circumstances (and thus have such code be @safe), shared in general still needs to be locked down in a manner similar to what this DIP is proposing. Any improvements that we later come up with would then just be laid on top of that.

- Jonathan M Davis

On Saturday, October 12, 2019 5:26:42 AM MDT Ola Fosheim Grøstad via Digitalmars-d wrote:
> I hope people try to sketch out what such libraries will look like before the semantics are decided on. Maybe an experimental branch would be appropriate.

The fact that we have thread-local in the language requires that the type system then have the concept of shared. So, there should be no question about that. As part of that, converting between thread-local and shared has to require casts (otherwise, the compiler can't guarantee that the thread-local stuff is actually thread-local). Those casts then have to be @system, because they potentially violate the compiler's guarantees about thread-local and shared data.

The question that then comes is what kinds of operations should be allowed on shared, and there are basically two approaches that can be taken:

1. Not allow read-write operations that aren't guaranteed to be atomic.

2. Ignore thread-safety and just let people do whatever they want to with shared other than implicitly convert between shared and thread-local (since that would violate the compiler guarantees about which data is shared and which is thread-local).

#1 allows us to segregate all of the code that has to deal with threading stuff in a manner that requires the programmer to get involved, because operating on shared in a way that isn't guarantee to be atomic then becomes @system. So, it becomes far easier to locate the pieces of code that need to be verified as thread-safe by the programmer, and you get actual compiler guarantees that any @safe code operating on shared variables doesn't have any threading problems. Only the @trusted code has to be examined.

Unlike #1, #2 would mean that you couldn't know that code operating on shared was thread-safe. You wouldn't have to do as much casting, because basic operations would work on shared data, but you'd then have to examine all code involving shared to determine whether what it was doing was thread-safe. And because many types are designed to be thread-local rather than shared, you'd still have to cast to thread-local in many cases, and that code would still have to be @system, because the programmer is potentially violating the compiler guarantees about thread-local when casting to or from shared.

So, while #2 would reduce the amount of casting required, it would do a far worse job segregating code that needed to be examined when dealing with threading issues, and it doesn't really remove the casting and the like that folks like to complain about with regards to shared. We were locked into that once we had shared as part of the type system. Languages like C++ can only get away with not having it because they ignore the issue. That means that in such languages, you can just operate on shared stuff as if it were thread-local, but the programmer then also has to worry about threading issues throughout the entire program, because the type system isn't helping them at all. Good programming practices help make that problem more manageable, but by having shared as part of the type system in D, the language and the compiler make the problem much more manageable by segregating the threading code.

Right now, we basically have #2. What this DIP is trying to do is move us to #1. How libraries should work is exactly the same in either case. It's just that with #1, the places where you operate on shared data in a manner which isn't guaranteed to be atomic, the compiler prevents you from doing it unless you  use core.atomic or have @system code with casts. Even if we have #2 and thus no such compiler errors, the code should still have been doing what #1 would have required, since if it doesn't, then it isn't thread-safe.

It may be that if we can come up with ways for the compiler to know that it's thread-safe to implicitly remove shared in a piece of code, libraries won't have to have as much @trusted code, but the low level semantics would still need to be the same, and any code that wasn't using whatever idiom or feature allowed the compiler to implicitly remove shared in a piece of code would still need to do what it would need to do right now, which is what it would need to do if this DIP were implemented.

All this DIP really does is force code that the compiler can't guarantee is atomic to use either core.atomic or be @system/@trusted. The idioms and design patterns involved with thread-safe code are the same either way.

- Jonathan M Davis

On Saturday, 12 October 2019 at 21:28:36 UTC, Jonathan M Davis wrote:
> The fact that we have thread-local in the language requires that the type system then have the concept of shared. So, there should be no question about that.

Well, my understanding from what you wrote is that the shared marker is 100% syntactical and has no semantic implications. In that case it can be done as a library meta programming construct (type wrapping). You just need a way to restrict application of the constructs  based on code structure (@safe). So one could establish a syntactical solution that is perfectly suitable for meta programming?

But you could probably do better by introducing an effectsbased typesystem.

Anyway, based on what you said shared does not have to be in the language and could be replaced by more expressive metaprogramming mechanisms. I think.

On Sat, Oct 12, 2019 at 2:20 AM Ola Fosheim Grøstad via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On Saturday, 12 October 2019 at 06:31:18 UTC, Jonathan M Davis wrote:
> > On Friday, October 11, 2019 10:58:29 PM MDT Walter Bright via Digitalmars-d wrote:
> >> On 10/2/2019 3:42 AM, Nicholas Wilson wrote:
>
> >> > It should be that shared memory access is disabled in all
> >> > contexts,
> >> > which must be worked around with casts,
> >> > which makes the function @system,
> >> > which must then be encapsulated with a @trusted interface in
> >> > order to
> >> > use in @safe code.
> >>
> >> Sounds right.
> >
> > This is pretty much what several of us were arguing for during the discussions about shared at dconf this year.
>
> How are you going to prove that @safe code does not retain nonshared-references after the lock has been released?
>
> How does his work with array elements/slices.

I think `scope` and `return` have something to say about avoiding escaping references. When the semantics are all working, we can produce such test code.

On Sat, Oct 12, 2019 at 4:30 AM Ola Fosheim Grøstad via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On Saturday, 12 October 2019 at 09:45:41 UTC, Jonathan M Davis wrote:
> > It's the same as with any @trusted code. It's up to the programmer to ensure that what the code is doing is actually @safe, and if the code isn't able to provide an API that is @safe, then it shouldn't be @trusted.
>
> So, I am trying to figure out what that API would have to look like and the only solution I can think of right away is that you have are doing a callback back into @safe code from the lock-holding @trusted code?
>
> So, if you have to obtain locks from multiple sources, it either won't work out, or you have to use some serious kludges, like a deep call chain:
>
> while(tryagain) {
>      trusted_code_obtains_lock_1-->
>         safe_code_callback1 calls->
>             trusted_code_obtains_lock2
>                 safe_code_callback2 processes and sets
> tryagain=false
> }
>
> Of course, that could lead to starvation.
>
> So to avoid all of that multi-threaded libraries have to have 2 APIs, one simple transactional wrapper-API for @safe code and another API for @trusted code that can be used to write a new wrapper that allows locking of multiple datastructures.
>
> I hope people try to sketch out what such libraries will look like before the semantics are decided on. Maybe an experimental branch would be appropriate.

Here's a rough draft of one such sort of tool I use all the time in shared-intensive code: https://gist.github.com/TurkeyMan/c16db7a0be312e9a0a2083f5f4a6efec

I just hacked that together the other night to do some experiments. I
have a toolbox like that which we use at work.
This code here feels fine to me in D, but it all depends on
restrictive shared to work.
There's nothing useful you can do with shared safely as it exists in
the language today.

On Saturday, 12 October 2019 at 21:28:36 UTC, Jonathan M Davis wrote:
>
> Right now, we basically have #2. What this DIP is trying to do is move us to #1. How libraries should work is exactly the same in either case. It's just that with #1, the places where you operate on shared data in a manner which isn't guaranteed to be atomic, the compiler prevents you from doing it unless you  use core.atomic or have @system code with casts. Even if we have #2 and thus no such compiler errors, the code should still have been doing what #1 would have required, since if it doesn't, then it isn't thread-safe.

With this DIP, shared integers/small types will be automatically atomic. For complex/large types, will you still be able to use them as before between threads and you have protect the type yourself at least for a transitional period?

"Atomic" here as I get it also mean atomically updating complex types. This usually means that you need to guard the operations with some kind of mutex. The compiler can of course detect this and issue a warning/error to the user which doesn't seem to be the scope of this DIP.

Correct me if I'm wrong but we have the following scenarios.
1. shared integer/simple type (size dependent?) -> automatically HW atomic operations
2. shared complex type -> write to any member must be protected with a mutex.
3. shared complex type -> read to any member must be protected with a mutex or read/write mutex allowing multiple reads.

The compiler is used the detect these scenarios so that the user doesn't forget protecting the shared types.

As I get it this DIP is just a baby step towards this bigger scope for the shared keyword.