On 08-Jul-2015 15:39, wobbles wrote:
> On Wednesday, 8 July 2015 at 12:21:22 UTC, Jonathan M Davis wrote:
>> On Wednesday, 8 July 2015 at 11:02:19 UTC, Márcio Martins wrote:
>>> [...]
>
>
> Interesting, so the main pain of using shared is the requirement to cast
> away the shared whenever you want to work on the data in a synchronized
> block.
>
> Is there any links do you know to the old conversations on what
> solutions are there for this?
>
> My first thought is using the 'with' keyword.
> shared int mySharedInt;
> synchronised(mutexObj) with (mySharedInt){
>      // any references to mySharedInt in this block are implicitly
> converted to non-shared
> }

It's good convention but still convention - who guarantees that the right mutex was locked? The locking protocol is outside of the competence of the compiler.

However there are synchronized _class_-es that might play nice with shared b/c very access to them is guarded by built-in mutex.

-- 
Dmitry Olshansky

On 08/07/15 15:08, Jonathan M Davis wrote:
> I know that there are a number of people who get frustrated with shared
> and using __gshared instead, but unless you fully understand what you're
> doing and how the language works, and you're _really_ careful, you're
> going to shoot yourself in the foot it subtle ways if you do that.
>
I guess my main issue with this statement is that I don't see how that is not the case when using shared.

Shachar

On Wednesday, 8 July 2015 at 12:51:02 UTC, Dmitry Olshansky wrote:
> On 08-Jul-2015 15:39, wobbles wrote:
>> On Wednesday, 8 July 2015 at 12:21:22 UTC, Jonathan M Davis wrote:
>>> On Wednesday, 8 July 2015 at 11:02:19 UTC, Márcio Martins wrote:
>>>> [...]
>>
>>
>> Interesting, so the main pain of using shared is the requirement to cast
>> away the shared whenever you want to work on the data in a synchronized
>> block.
>>
>> Is there any links do you know to the old conversations on what
>> solutions are there for this?
>>
>> My first thought is using the 'with' keyword.
>> shared int mySharedInt;
>> synchronised(mutexObj) with (mySharedInt){
>>      // any references to mySharedInt in this block are implicitly
>> converted to non-shared
>> }
>
> It's good convention but still convention - who guarantees that the right mutex was locked? The locking protocol is outside of the competence of the compiler.

Yes. That's the problem, and it would be great if we could find a solution for it. But as annoying as it is, we're still better off than what you get with C++.

> However there are synchronized _class_-es that might play nice with shared b/c very access to them is guarded by built-in mutex.

Yes. If synchronized classes were actually implemented, then that provides a compiler guaranteed safe way to strip off the outer layered of shared, but it only strips off the outer layer (since it can't guarantee that stripping off anymore would be safe), which would severely limit its usefulness, and it's rather clunky and verbose to have to declare whole classes just to operate on shared data. So, thus far, it's the best that we've come up with for safely casting away shared in a compiler-guaranteed way, but it's still not all that great, and it's not even implemented.

So, ultimately, even if we finally do get synchronized classes, I expect that there will be a fair bit of code that's going to have to rely on the programmer to correctly and safely cast away shared to operate on data, as unappealing as that may be.

Regardless, until we have synchronized classes or another solution which does something similar, the idiom that I described is pretty much what we have to do, even if it unfortunately relies on the programmer following convention and being careful. But it is basically what you have to do in languages like C++ anyway except that casting is involved, and the portion of the code where it's necessary is a lot clearer thanks to the fact that shared objects are explicitly shared and how shared doesn't allow you to do much.

- Jonathan M Davis

On Wednesday, 8 July 2015 at 13:26:53 UTC, Shachar Shemesh wrote:
> On 08/07/15 15:08, Jonathan M Davis wrote:
>> I know that there are a number of people who get frustrated with shared
>> and using __gshared instead, but unless you fully understand what you're
>> doing and how the language works, and you're _really_ careful, you're
>> going to shoot yourself in the foot it subtle ways if you do that.
>>
> I guess my main issue with this statement is that I don't see how that is not the case when using shared.

Because unless you cast away shared, you're prevented from doing much of anything to the object, and the compiler clearly indicates which objects are shared, so the code that has to worry about getting locks right and dealing with casting away shared correctly is clearly marked and segregated from the rest of the program, unlike with a language like C++ or Java where _everything_ is shared, and you have no idea which objects are actually shared across threads and which are thread-local.

What we have is uglier than we'd like, but that ugliness highlights the small portion of code where you actually have to deal with synchronization and threading issues so that if you do have a problem with it, you have a very small amount of code to dig through to figure out how you screwed it up.

- Jonathan M Davis

On Wednesday, 8 July 2015 at 14:00:43 UTC, Jonathan M Davis wrote:
> On Wednesday, 8 July 2015 at 13:26:53 UTC, Shachar Shemesh wrote:
>> On 08/07/15 15:08, Jonathan M Davis wrote:
>>> I know that there are a number of people who get frustrated with shared
>>> and using __gshared instead, but unless you fully understand what you're
>>> doing and how the language works, and you're _really_ careful, you're
>>> going to shoot yourself in the foot it subtle ways if you do that.
>>>
>> I guess my main issue with this statement is that I don't see how that is not the case when using shared.
>
> Because unless you cast away shared, you're prevented from doing much of anything to the object, and the compiler clearly indicates which objects are shared, so the code that has to worry about getting locks right and dealing with casting away shared correctly is clearly marked and segregated from the rest of the program, unlike with a language like C++ or Java where _everything_ is shared, and you have no idea which objects are actually shared across threads and which are thread-local.
>
> What we have is uglier than we'd like, but that ugliness highlights the small portion of code where you actually have to deal with synchronization and threading issues so that if you do have a problem with it, you have a very small amount of code to dig through to figure out how you screwed it up.
>
> - Jonathan M Davis

I think a good way to avoid this extra annoyance would be to have shared be implicitly convertible to non-shared, or actually, shared should just be ignored inside synchronized blocks, essentially the net result is that the cast is done implicitly at the beginning of the block.

To solve the wrong mutex problem that Dmitry mentioned, perhaps a declarative approach could be used? I wouldn't mind the extra syntax, as it also provides documentation by giving clarity into which mutexes guard what data. With the implicit stripping/ignoring of shared it would become very succinct as well.

```
Mutex moomtx;
Mytex cheesemtx;

shared(moomtx) int[] moo;
shared(cheesemtx) float cheese;

or perhaps with UDA:

@lock(globalmtx, moomtx) shared int moo; // either globalmtx or moomtx must be locked


synchronized(moomtx) {
   moomtx = []; // ok
   cheese = 1.61803f; // error: mutex moomtx is not locked, even though we are inside a synchronized block

   moo.sort(); // shared automatically ignored
}

// ...
moo.sort(); // error: mutex moomtx is not locked
```

On Wednesday, 8 July 2015 at 14:29:56 UTC, Márcio Martins wrote:
> I think a good way to avoid this extra annoyance would be to have shared be implicitly convertible to non-shared, or actually, shared should just be ignored inside synchronized blocks, essentially the net result is that the cast is done implicitly at the beginning of the block.
>
> To solve the wrong mutex problem that Dmitry mentioned, perhaps a declarative approach could be used? I wouldn't mind the extra syntax, as it also provides documentation by giving clarity into which mutexes guard what data. With the implicit stripping/ignoring of shared it would become very succinct as well.

Synchronized classes would be able to remove the outer layer of shared, because they can actually guarantee that no one else has access to their member variables (e.g. it would be illegal to make them public). So, when it locks the mutex on a member function call, it can actually guarantee that stripping that outer layer of shared is safe. But it can only strip away the outer layer, because it can't make any guarantees beyond that.

For shared to work like you're suggesting, the compiler would essentially need to make it so that doing _anything_ to a shared object when the mutex wasn't locked would be illegal, and I don't know how feasible that really is. And even if it could do it, all it would be doing would be the same as a synchronized class and stripping away the outer layer of shared. But even then, it wouldn't be enough, because it has no way of stopping you from assigning that partially-unshared object to something and letting it escape, whereas with a synchronized class, everything you're doing is inside of a member function, so it's encapsulated, allowing the compiler to make better guarantees about what you are or aren't doing with the object. So, for what you're proposing, you pretty much might as well just create a synchronized class which uses opDispatch to forward calls to the member variable. It's slightly more verbose at the declaration site but less verbose wherever you use the object. And regardless, it doesn't solve the problem that all that you can strip away of shared is the outer layer. I'm not sure that it's possible to do any more than that with compiler guarantees.

Another thing to consider is that you might need to lock multiple objects with the same mutex or lock multiple mutexes when accessing an object, in which case, what you're proposing is definitely worse than what we'd get from synchronized classes, since they naturally lock multiple items at once (though they don't really cover the case where multiple mutexes need to be involved except that because all of their functionality is encapsulated, it's easier to put the extra locking where it needs to be without missing it).

Maybe we need to do something like you're suggesting, but really, it doesn't seem like it's improving particularly on synchronized classes.

- Jonathan M Davis

On 08-Jul-2015 16:57, Jonathan M Davis wrote:
> On Wednesday, 8 July 2015 at 12:51:02 UTC, Dmitry Olshansky wrote:
>> On 08-Jul-2015 15:39, wobbles wrote:

>
> So, ultimately, even if we finally do get synchronized classes, I expect
> that there will be a fair bit of code that's going to have to rely on
> the programmer to correctly and safely cast away shared to operate on
> data, as unappealing as that may be.
>

I definetly seen synchronized classes in TDPL years ago. They ought to work today or am I missing something?


-- 
Dmitry Olshansky

On 7/8/15 5:20 AM, wobbles wrote:
> After reading the recent "Lessons Learned" article [1], and reading a
> few comments on the thread, there was a mention of using __gshared over
> shared.
>

I don't see any full answers to this so:

> What exactly is the difference here?

__gshared just puts the data in global segment, but does NOT alter the type. shared DOES alter the type:

__gshared int x1;
shared int x2;

pragma(msg, typeof(x1)); // int
pragma(msg, typeof(x2)); // shared(int)

Why is this important? Because you can overload on shared data to do special things (i.e. reject pointers to shared data, or use mutex locks around only truly shared data). __gshared data is only strictly a storage class, so you cannot do anything different with e.g. &x1 as you could with an address to a normal thread-local int.

As has been mentioned, __gshared data more accurately represents how C treats global data, but technically, you could use C to access shared variables. Both would have to be tagged with extern(C).

> Is there plans to 'converge' them at some point?

No, they are different concepts.

-Steve

On Wednesday, 8 July 2015 at 18:05:13 UTC, Dmitry Olshansky wrote:
> I definetly seen synchronized classes in TDPL years ago. They ought to work today or am I missing something?

They're described in TDPL, but they've never been implemented. Instead, we have synchronized functions like in Java (which TDPL specifically talks about being a bad idea). With a synchronized class, _all_ functions in a class would be synchronized, and a variety of restrictions are placed on the class so that it can make implicitly remove the outer layed or shared on its member variables inside its member functions. With synchronized functions, all they do is lock when you enter them an unlock when you exit them and share the mutex with the other synchronized functions in the class, but they add no other abilities or guarantees.

So, arguably, we should implement synchronized classes as described in TDPL, and that would certainly help with the shared situation, but for whatever reason, no one has ever implemented them.

- Jonathan M Davis

On Wednesday, 8 July 2015 at 12:08:37 UTC, Jonathan M Davis wrote:
> By using __gshared, you're throwing away the compiler's help, and it's _much_ more likely that you're going to write code which causes the compiler to generate incorrect machine code, because it's assuming that an object is thread-local when it's not.
>
> Generally what you have to do with shared is lock on a mutex, cast away shared on the object you want to operate on, do whatever you're going to do with it, and then release the lock after there are no more thread-local references to the shared object. And that's basically what you normally should be doing in C++ code except that you don't have to cast away shared, because C++ doesn't have it.
>
> I know that there are a number of people who get frustrated with shared and using __gshared instead, but unless you fully understand what you're doing and how the language works, and you're _really_ careful, you're going to shoot yourself in the foot it subtle ways if you do that.
>
> - Jonathan M Davis

Amen