On 2012-11-14 22:06, Walter Bright wrote:

> I hate to repeat myself, but:
>
> Thread 1:
>      1. create shared object
>      2. pass reference to that object to Thread 2
>      3. destroy object
>
> Thread 2:
>      1. manipulate that object

Why would the object be destroyed if there's still a reference to it? If the object is manually destroyed I don't see what threads have to do with it since you can do the same thing in a single thread application.

-- 
/Jacob Carlborg

On 15 November 2012 04:30, Andrei Alexandrescu < SeeWebsiteForEmail@erdani.org> wrote:

> On 11/11/12 6:30 PM, Walter Bright wrote:
>
>> 1. ensure single threaded access by aquiring a mutex
>> 2. cast away shared
>> 3. operate on the data
>> 4. cast back to shared
>> 5. release the mutex
>>
>
> This is very different from how I view we should do things (and how we actually agreed to do things and how I wrote in TDPL).
>
> I can't believe I need to restart this on a cold cache.


The pattern Walter describes is primitive and useful, I'd like to see
shared assist to that end (see my previous post).
You can endeavour to do any other fancy stuff you like, but until some
distant future when it's actually done, then proven and well supported,
I'll keep doing this.

Not to repeat my prev post... but in reply to Walter's take on it, it would be interesting if 'shared' just added implicit lock()/unlock() methods to do the mutex acquisition and then remove the cast requirement, but have the language runtime assert that the object is locked whenever it is accessed (this guarantees the safety in a more useful way, the casts are really annying). I can't imagine a simpler and more immediately useful solution.

In fact, it's a reasonably small step to this being possible with user-defined attributes. Although attributes have no current mechanism to add a mutex, and lock/unlock methods to the object being attributed (like is possible in Java/C#), but maybe it's not a huge leap.

On 2012-11-15 10:22, Manu wrote:

> Not to repeat my prev post... but in reply to Walter's take on it, it
> would be interesting if 'shared' just added implicit lock()/unlock()
> methods to do the mutex acquisition and then remove the cast
> requirement, but have the language runtime assert that the object is
> locked whenever it is accessed (this guarantees the safety in a more
> useful way, the casts are really annying). I can't imagine a simpler and
> more immediately useful solution.

How about implementing a library function, something like this:

shared int i;

lock(i, (x) {
    // operate on x
});

* "lock" will acquire a lock
* Cast away shared for "i"
* Call the delegate with the now plain "int"
* Release the lock

http://pastebin.com/tfQ12nJB

-- 
/Jacob Carlborg

On 15 November 2012 12:14, Jacob Carlborg <doob@me.com> wrote:

> On 2012-11-15 10:22, Manu wrote:
>
>  Not to repeat my prev post... but in reply to Walter's take on it, it
>> would be interesting if 'shared' just added implicit lock()/unlock() methods to do the mutex acquisition and then remove the cast requirement, but have the language runtime assert that the object is locked whenever it is accessed (this guarantees the safety in a more useful way, the casts are really annying). I can't imagine a simpler and more immediately useful solution.
>>
>
> How about implementing a library function, something like this:
>
> shared int i;
>
> lock(i, (x) {
>     // operate on x
> });
>
> * "lock" will acquire a lock
> * Cast away shared for "i"
> * Call the delegate with the now plain "int"
> * Release the lock
>
> http://pastebin.com/tfQ12nJB


Interesting concept. Nice idea, could certainly be useful, but it doesn't
address the problem as directly as my suggestion.
There are still many problem situations, for instance, any time a template
is involved. The template doesn't know to do that internally, but under my
proposal, you lock it prior to the workload, and then the template works as
expected. Templates won't just break and fail whenever shared is involved,
because assignments would be legal. They'll just assert that the thing is
locked at the time, which is the programmers responsibility to ensure.

On Thu, 15 Nov 2012 04:33:20 -0000, Michel Fortin <michel.fortin@michelf.ca> wrote:

> On 2012-11-15 02:51:13 +0000, "Jonathan M Davis" <jmdavisProg@gmx.com> said:
>
>> I have no idea what we want to do about this situation though. Regardless of
>> what we do with memory barriers and the like, it has no impact on whether
>> casts are required.
>
> Let me restate and extend that idea to atomic operations. Declare a variable using the synchronized storage class and it automatically get a mutex:
>
> 	synchronized int i; // declaration
>
> 	i++; // error, variable shared
>
> 	synchronized (i)
> 		i++; // fine, variable is thread-local inside synchronized block
>
> Synchronized here is some kind of storage class causing two things: a mutex is attached to the variable declaration, and the type of the variable is made shared. The variable being shared, you can't access it directly. But a synchronized statement will make the variable non-shared within its bounds.
>
> Now, if you want a custom mutex class, write it like this:
>
> 	synchronized(SpinLock) int i;
>
> 	synchronized(i)
> 	{
> 		// implicit: i.mutexof.lock();
> 		// implicit: scope (exit) i.mutexof.unlock();
> 		i++;
> 	}
>
> If you want to declare the mutex separately, you could do it by specifying a variable instead of a type in the variable declaration:
>
> 	Mutex m;
> 	synchronized(m) int i;
> 	
> 	synchronized(i)
> 	{
> 		// implicit: m.lock();
> 		// implicit: scope (exit) m.unlock();
> 		i++;
> 	}
>
> Also, if you have a read-write mutex and only need read access, you could declare that you only need read access using const:
>
> 	synchronized(RWMutex) int i;
>
> 	synchronized(const i)
> 	{
> 		// implicit: i.mutexof.constLock();
> 		// implicit: scope (exit) i.mutexof.constUnlock();
> 		i++; // error, i is const
> 	}
>
> And finally, if you want to use atomic operations, declare it this way:
>
> 	synchronized(Atomic) int i;
>
> You can't really synchronize on something protected by Atomic:
>
> 	syncronized(i) // cannot make sycnronized block, no lock/unlock method in Atomic
> 	{}
>
> But you can call operators on it while synchronized, it works for anything implemented by Atomic:
>
> 	synchronized(i)++; // implicit: Atomic.opUnary!"++"(i);
>
> Because the policy object is associated with the variable declaration, when locking the mutex you need direct access to the original variable, or an alias to it. Same for performing atomic operations. You can't pass a reference to some function and have that function perform the locking. If that's a problem it can be avoided by having a way to pass the mutex to the function, or by passing an alias to a template.

+1

I suggested something similar as did Sönke:
http://forum.dlang.org/thread/k7orpj$1tt5$1@digitalmars.com?page=2#post-op.wnnuiio554xghj:40puck.auriga.bhead.co.uk

According to deadalnix the compiler magic I suggested to add the mutex isn't possible:
http://forum.dlang.org/thread/k7orpj$1tt5$1@digitalmars.com?page=3#post-k7qsb5:242gqk:241:40digitalmars.com

Most of our ideas can be implemented with a wrapper template containing the sync object (mutex, etc).

So... my feeling is that the best solution for "shared", ignoring the memory barrier aspect which I would relegate to a different feature and solve a different way, is..

1. Remove the existing mutex from object.
2. Require that all objects passed to synchronized() {} statements implement a synchable(*) interface
3. Design a Shared(*) wrapper template/struct that contains a mutex and implements synchable(*)
4. Design a Shared(*) base class which contains a mutex and implements synchable(*)

Then we design classes which are always shared using the base class and we wrap other objects we want to share in Shared() and use them in synchronized statements.

This would then relegate any builtin "shared" statement to be solely a storage class which makes the object global and not thread local.

(*) names up for debate

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

On Wednesday, November 14, 2012 20:32:35 Andrei Alexandrescu wrote:
> TDPL 13.14 explains that inside synchronized classes, top-level shared is automatically lifted.

Then it's doing the casting for you. I suppose that that's an argument that using synchronized classes when dealing with shared is the way to go (which IIRC TDPL does argue), but that only applies to classes, and there are plenty of cases (maybe even the majority) where it's built-in types like arrays or AAs which people are trying to share, and synchronized classes won't help them there unless they create wrapper types. And explicit casting will be required for them. And of course, anyone wanting to use mutexes or synchronized blocks will have to use explicit casts regardless of what they're protecting, because it won't be inside a synchronized class. So, while synchronized classes make dealing with classes nicer, they only handle a very specific portion of  what might be used with shared.

In any case, I clearly need to reread TDPL's threading stuff (and maybe the whole book). It's been a while since I read it, and I'm getting rusty on the details.

By the way, speaking of synchronized classes, as I understand it, they're still broken with regards to TDPL in that synchronized is still used on functions rather than classes like TDPL describes. So, they aren't currently a solution regardless of what the language actual design is supposed to be. Obviously, that should be fixed though.

- Jonathan M Davis

On Thursday, November 15, 2012 11:22:30 Manu wrote:
> Not to repeat my prev post... but in reply to Walter's take on it, it would be interesting if 'shared' just added implicit lock()/unlock() methods to do the mutex acquisition and then remove the cast requirement, but have the language runtime assert that the object is locked whenever it is accessed (this guarantees the safety in a more useful way, the casts are really annying). I can't imagine a simpler and more immediately useful solution.
> 
> In fact, it's a reasonably small step to this being possible with user-defined attributes. Although attributes have no current mechanism to add a mutex, and lock/unlock methods to the object being attributed (like is possible in Java/C#), but maybe it's not a huge leap.

1. It wouldn't stop you from needing to cast away shared at all, because without casting away shared, you wouldn't be able to pass it to anything, because the types would differ. Even if you were arguing that doing something like

void foo(C c) {...}
shared c = new C;
foo(c); //no cast required, lock automatically taken

it wouldn't work, because then foo could wile away a reference to c somewhere, and the type system would have no way of knowing that it was a shared variable that was being wiled away as opposed to a thread-local one, which means that it'll likely generate incorrect code. That can happen with the cast as well, but at least in that case, you're forced to be explicit about it, and it's automatically @system. If it's done for you, it'll be easy to miss and screw up.

2. It's often the case that you need to lock/unlock groups of stuff together such that locking specific variables is of often of limited use and would just introduce pointless extra locks when dealing with multiple variables. It would also increase the risk of deadlocks, because you wouldn't have much - if any - control over what order locks were acquired in when dealing with multiple shared variables.

- Jonathan M Davis

On Thursday, November 15, 2012 10:22:22 Jacob Carlborg wrote:
> On 2012-11-14 22:06, Walter Bright wrote:
> > I hate to repeat myself, but:
> > 
> > Thread 1:
> >      1. create shared object
> >      2. pass reference to that object to Thread 2
> >      3. destroy object
> > 
> > Thread 2:
> >      1. manipulate that object
> 
> Why would the object be destroyed if there's still a reference to it? If the object is manually destroyed I don't see what threads have to do with it since you can do the same thing in a single thread application.

Yeah. If the reference passed across were shared, then the runtime should see it as having multiple references, and if it's _not_ shared, that means that you cast shared away (unsafe, since it's a cast) and passed it across threads without making sure that it was the only reference on the original thread. In that case, you shot yourself in the foot by using an @system construct (casting) and not getting it right. I don't see why the runtime would have to worry about that.

Unless the problem is that the object is a value type, so when it goes away on the first thread, it _has_ to be destroyed? If that's the case, then it's a pointer that was passed across rather than a reference, and then you've effectively done the same thing as returning a pointer to a local variable, which is @system and again only happens if you're getting @system wrong, which the compiler generally doesn't protect you from beyond giving you an error in the few cases where it can determine for certain that what you're doing is wrong (which is a fairly limited portion of the time).

So, as far as I can see - unless I'm just totally missing something here - either you're dealing with shared objects on the heap here, in which case, the object shouldn't be destroyed on the first thread unless you do it manually (in which case, you're doing something stupid in @system code), or you're dealing with passing pointers to shared value types across threads, which is essentially the equivalent of escaping a pointer to a local variable (in which case, you're doing something stupid in @system code). In either case, it's you're doing something stupid in @system code, and I don't see why the runtime would have to worry about it. You shot yourself in the foot by incorrectly using @system code. If you want protection agains that, then don't use @system code.

- Jonathan M Davis

Am 15.11.2012 12:48, schrieb Jonathan M Davis:
>
> Yeah. If the reference passed across were shared, then the runtime should see
> it as having multiple references, and if it's _not_ shared, that means that
> you cast shared away (unsafe, since it's a cast) and passed it across threads
> without making sure that it was the only reference on the original thread. In
> that case, you shot yourself in the foot by using an @system construct
> (casting) and not getting it right. I don't see why the runtime would have to
> worry about that.
>
> Unless the problem is that the object is a value type, so when it goes away on
> the first thread, it _has_ to be destroyed? If that's the case, then it's a
> pointer that was passed across rather than a reference, and then you've
> effectively done the same thing as returning a pointer to a local variable,
> which is @system and again only happens if you're getting @system wrong, which
> the compiler generally doesn't protect you from beyond giving you an error in
> the few cases where it can determine for certain that what you're doing is
> wrong (which is a fairly limited portion of the time).
>
> So, as far as I can see - unless I'm just totally missing something here -
> either you're dealing with shared objects on the heap here, in which case, the
> object shouldn't be destroyed on the first thread unless you do it manually (in
> which case, you're doing something stupid in @system code), or you're dealing
> with passing pointers to shared value types across threads, which is
> essentially the equivalent of escaping a pointer to a local variable (in which
> case, you're doing something stupid in @system code). In either case, it's
> you're doing something stupid in @system code, and I don't see why the runtime
> would have to worry about it. You shot yourself in the foot by incorrectly
> using @system code. If you want protection agains that, then don't use @system
> code.
>
> - Jonathan M Davis
>

Thank you, thats exatcly how I'm thinking too. And because of this it makes absolutley no sense to me to disallow the destruction of a shared struct, if it is allocated on the stack or as a global. If it is allocated on the heap you can't destory it manually anyway because delete is deprecated.

And for exatcly this reason I wanted a code example from Walter. Because just listing a few bullet points does not make a real world use case.

Kind Regards
Benjamin Thaut

On 15 November 2012 13:38, Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> On Thursday, November 15, 2012 11:22:30 Manu wrote:
> > Not to repeat my prev post... but in reply to Walter's take on it, it
> would
> > be interesting if 'shared' just added implicit lock()/unlock() methods to
> > do the mutex acquisition and then remove the cast requirement, but have
> the
> > language runtime assert that the object is locked whenever it is accessed (this guarantees the safety in a more useful way, the casts are really annying). I can't imagine a simpler and more immediately useful solution.
> >
> > In fact, it's a reasonably small step to this being possible with user-defined attributes. Although attributes have no current mechanism to add a mutex, and lock/unlock methods to the object being attributed (like is possible in Java/C#), but maybe it's not a huge leap.
>
> 1. It wouldn't stop you from needing to cast away shared at all, because
> without casting away shared, you wouldn't be able to pass it to anything,
> because the types would differ. Even if you were arguing that doing
> something
> like
>
> void foo(C c) {...}
> shared c = new C;
> foo(c); //no cast required, lock automatically taken
>
> it wouldn't work, because then foo could wile away a reference to c
> somewhere,
> and the type system would have no way of knowing that it was a shared
> variable
> that was being wiled away as opposed to a thread-local one, which means
> that
> it'll likely generate incorrect code. That can happen with the cast as
> well,
> but at least in that case, you're forced to be explicit about it, and it's
> automatically @system. If it's done for you, it'll be easy to miss and
> screw
> up.
>

I don't really see the difference, other than, as you say, the cast is
explicit.
Obviously the possibility for the situation you describe exists, it's
equally possible with the cast, except this way, the usage pattern is made
more convenient, the user has a convenient way to control the locks and
most importantly, it would work with templates.
That said, this sounds like another perfect application of 'scope'. Perhaps
only scope parameters can receive a locked, shared thing... that would
mechanically protect you against escape.

2. It's often the case that you need to lock/unlock groups of stuff together
> such that locking specific variables is of often of limited use and would
> just
> introduce pointless extra locks when dealing with multiple variables. It
> would
> also increase the risk of deadlocks, because you wouldn't have much - if
> any -
> control over what order locks were acquired in when dealing with multiple
> shared variables.


Your fear is precisely the state we're in now, except it puts all the work
on the user to create and use the synchronisation objects, and also to
assert that things are locked when they are accessed.
I'm just suggesting some reasonably simple change that would make the
situation more usable and safer immediately, short of waiting for all these
fantastic designs being discussed having time to simmer and manifest.

Perhaps a usage mechanism could be more like:
shared int x, y, z;
synchronised with(x, y, z)
{
  // do work with x, y, z, all locked together.
}