Jonathan M Davis 
Posted in reply to FeepingCreature
| On Wednesday, October 31, 2018 2:08:58 AM MDT FeepingCreature via Digitalmars-d wrote:
> On Wednesday, 31 October 2018 at 07:57:59 UTC, Jonathan M Davis
> > Manual casting is required if you're actually using shared with mutexes or synchronized. If you're using __gshared, it's not, but that's only because you're lying to the compiler about whether the variables are shared are not, which means that you're circumventing the type system, and you're risking subtle bugs in your programs, because the guarantees that the type system is suppose make aren't actually in place.
>
> This does not match what I'm actually seeing in practice. If this is how it is, then this admonishment should be written in the D style guide in big, bolded letters.
Probably. There are places in the documentation that talk about it, but it's almost certainly not clear enough given how many folks keep using it. Certainly, far too many folks run to __gshared when they get annoyed with shared.
In general, we've done a terrible job messaging how shared works, in part because it's been 95% complete for ages, and it's never been a priority. In general, what shared must do (and _mostly_ does) is guarantee that no operations that are not thread-safe are illegal. Earlier on, there was a larger push to find ways to get the compiler to then do stuff for you to make stuff thread-safe (such as introducing write barriers), and Andrei tends to prefer that approach, but that approach is hard to do (if possible at all), and pieces that we _can_ do (like write barriers) tend to be inefficient. Walter prefers the approach of just making more operations illegal (which unfortunately, then tends to require more casts and more @trusted code). So, some stuff has been left in limbo that really shouldn't have been left in limbo, and so we have shared - which works - but where some operations aren't actually thread-safe but are still legal (hence why I say it's 95% complete). And that still needs to be sorted out. And since it hasn't been a priority, and it's perpetually mostly finished, you don't tend to get much in the way of stuff like documentation write-ups about it. That really should be fixed.
> Generally we try to write cast-free code if possible, so casting away shared feels wrong. That's not really an argument, of course, but to me semantic casting indicates "no compiler, you are mistaken about the properties of this data." The compiler should never *require* casting.
In the case of shared, it does require casting, and there really isn't a good way around it. The two major exceptions are atomics and synchronized classes (though those aren't fully implemented). synchronized classes still do the casting; they're just able to cast away shared on the outer layer implicitly. It would be fantastic to be able to do it implicitly in more cases, but it can only be done implicitly in cases where the compiler can actually guarantee that it's thread-safe, and that's really, really hard to do. synchronized classes are the best proposal that we've had for a mechanism for doing so. More would likely be possible with a more complicated ownership model, but without that, the compiler simply doesn't have enough information. It needs to know stuff like that a mutex is associated with a particular variable or set of variables, that it's definitely locked in a section of code, and that it's absolutely impossible that there are any other references to that data which are not currently protected by that mutex. The biggest obstacle there is probably the bit about there being no references not currently protected. Without a real ownership model, that just doesn't work. Maybe someone will come up with a bright idea, but everything I've seen at best has subtle holes in it, and ultimately, it comes down to basically writing the same kind of code that you'd write in C/C++ except that you need a few extra casts within the sections of code where the shared variables are protected by a locked mutex.
> > Your synchronized field proposal is basically just a variation of synchronized classes with the extra caveat that somehow mutable references to the member variables can't escape, and for some reason, not everything in the class has to be synchronized.
>
> Everything in the class has to be synchronized. I'm not sure where I said differently.
You were talking about marking individual fields and variables as synchronized rather than simply marking the entire class as synchronized.
> > It's like you're trying to have synchronized classes without talking about shared. But you can't take shared out of the mix. Per the type system, anything that isn't shared is considered thread-local by the compiler, and trying to do anything like this without shared would be a serious violation of the type system and the compiler's guarantees.
>
> Again, imo this belongs in either the spec or the DStyle guide in big bold letters.
>
> > Instead of talking about safely casting away the outer layer of shared inside the class, you're talking about preventing escaping. synchronized classes as described in TDPL already guaranteed that the outer layer can't escape, since it lives directly in the class. What they can't guarantee is that the rest can't escape, and the language doesn't provide any way for such a guarantee.
>
> Not sure what you mean by "outer layer" here or this entire paragraph.
The part directly embedded in the class and which therefore cannot possibly escape. Unlike regular classes, synchronized classes do not allow direct access to their member variables - even from anything else in the same module - in order to be able to provide that guarantee. If you don't have your own copy of TDPL, I'd suggest reading the concurrency chapter which is available for free online, since it talks about synchronized classes:
http://www.informit.com/articles/article.aspx?p=1609144
Unfortunately, they're only partially implemented. IIRC, restricting access to the member variables has been implemented, but beyond that, right now they're mostly just synchronized functions like in Java.
> > And my point still stands that anything that we do with regards to thread safety must be built on top of shared, because that is the building block of sharing data across threads in D. Whether you like the specifics of shared or not, the idea that the compiler can rely on the assumption that data is _not_ shared across threads unless it's marked as such is a key feature of D. And attempting to use __gshared to get around it is just shooting yourself in the foot.
>
> And yet, there's no way to even *state* that the Thread constructor must not take a delegate that can access unshared data, and none of the current proposals list one. Should we demand that only shared data can be passed to new threads? That's going to limit their usefulness. With synchronized, we can deprecate the delegate constructor, then pass the thread data as synchronized objects to a pure function, thereby actually guaranteeing thread safety.
Well, in principle if you're passing data across threads, it should be shared while it's passed across threads, and the type system requires that. That's why you have to cast to immutable when passing reference types with std.concurrency (shared would work too except for an issue with std.variant's implementation). You can cast to thread-local again on the other side, but to get it across, it needs to be typed as shared / immutable, because that's what it really is at that point. And what you're doing is inherently @system, hence why the casting is required. It's up to the programmer to ensure that no references to the data remain on the original thread. If we had a full-own ownership model in the type system, then we might be able to avoid that, but we don't so we can't.
Thread is a bit different, but the concept is basically the same. Conceptually, it's being passed off to another thread, so it really should be shared or immutable, but C doesn't actually _have_ shared (hence __gshared), so everything involved _has_ to be @trusted internally. Looking at Thread's constructor, having it take a delegate probably isn't a problem, but what _is_ a problem is the fact that it's @safe. I don't know how we can reasonably fix that without breaking code, but it's definitely a problem. All data passed through that delegate must either be shared or it must be the only reference to that data so that when it ends up on that new thread, and it's thread-local, no other thread has reference to it, and it really is properly thread-local. That constructor can't guarantee that. It's up to the programmer to guarantee that. So, it needs to be @system.
If we could somehow require that the delegate only accepted shared data, then we could reasonably make the constructor @safe, but since AFAIK, that's not possible, it needs to be @system.
- Jonathan M Davis
|
Permalink
Reply
