On 10/10/13 19:31, Jonathan M Davis wrote:
> I'm honestly surprised that Andrei is rejecting the idea of casting to/from
> shared or immutable being normal given how it's required by our current
> concurrency model. And changing that would be a _big_ change.

I'm starting to incline towards the view that type qualifications of _any_ kind become problematic once you start working with any types other than built-in, and not just in the context of concurrency.  See e.g.:
http://d.puremagic.com/issues/show_bug.cgi?id=11148
http://d.puremagic.com/issues/show_bug.cgi?id=11188

I'd really appreciate advice on how to handle issues like these, because it's becoming a serious obstacle to my work on std.rational.

On Oct 10, 2013, at 10:23 AM, Joseph Rushton Wakeling <joseph.wakeling@webdrake.net> wrote:

> On 09/10/13 06:25, Andrei Alexandrescu wrote:
>> The way I see it we must devise a robust solution to that, NOT consider the state of the art immutable (heh, a pun).
> 
> Must say I have had a miserable experience with immutability and any kind of complex data structure, particularly when concurrency is involved.

As long as the reference itself can be reassigned (tail-immutable, I suppose) I think immutable is occasionally quite useful for complex data structures.  It basically formalizes the RCU (read-copy-update) approach to wait-free concurrency.  I'd tend to use this most often for global data structures built up on app start, and updated rarely to never as the program runs.

On 10/10/13 19:39, Sean Kelly wrote:
> As long as the reference itself can be reassigned (tail-immutable, I suppose) I think immutable is occasionally quite useful for complex data structures.  It basically formalizes the RCU (read-copy-update) approach to wait-free concurrency.  I'd tend to use this most often for global data structures built up on app start, and updated rarely to never as the program runs.

This kind of stuff is outside my experience, so if you'd like to offer a more detailed explanation/example, I'd be very grateful :-)

On Oct 10, 2013, at 10:36 AM, Joseph Rushton Wakeling <joseph.wakeling@webdrake.net> wrote:

> On 10/10/13 19:31, Jonathan M Davis wrote:
>> I'm honestly surprised that Andrei is rejecting the idea of casting to/from shared or immutable being normal given how it's required by our current concurrency model. And changing that would be a _big_ change.
> 
> I'm starting to incline towards the view that type qualifications of _any_ kind become problematic once you start working with any types other than built-in, and not just in the context of concurrency.  See e.g.:
> http://d.puremagic.com/issues/show_bug.cgi?id=11148
> http://d.puremagic.com/issues/show_bug.cgi?id=11188

I'm inclined to agree about shared.  But I see this largely as more encouragement to keep data thread-local in D.  If we can clean up move semantics via std.concurrency, I would be reasonably happy with data sharing in D.

As for const / immutable, I guess I don't see this as such an issue because I've been dealing with it in C++ for so long.  You either have to commit 100% to using const attributes or not use them at all.  Anything in between is fraught with problems.

On Thursday, October 10, 2013 10:27:24 Sean Kelly wrote:
> On Oct 9, 2013, at 9:24 PM, Jonathan M Davis <jmdavisProg@gmx.com> wrote:
> > And given that std.concurrency requires casting to and from shared or immutable in order to pass objects across threads, it seems ilke most of D's concurrency model requires casting to and/or from shared or immutable.
> std.concurrency won't be this way forever though. We could fake move semantics with something like assumeUnique!T, so send could be modified to accept a non-shared class that's marked as Unique.

I take it that you mean something other than std.exception.assumeUnique which simply casts to immutable? All that std.exception.assumeUnique does for you over casting is document why the cast is happening.

If you're talking about creating a wrapper type which indicates that the object is unique, I'd still expect that the casting would have to be happening underneath the hood when the object was passed (though then for better or worse, it would be encapsulated). And unless the objecte were always in that Unique wrapper, the programmer would still have to be promising that the object was actually unique and not being shared across threads rather than the type system doing it, in which case, I don't see much gain over simply casting. And if it's always in the wrapper, then you're in a similar boat to shared or immutable in that it's not the correct type.

I expect that there are nuances in what you're suggesting that I don't grasp at the moment, but as far as I can tell, the type system fundamentally requires a cast when passing objects across threads. It's just a question of whether that cast is hidden or not, and depending on how you hide it, I think that there's a real risk of the situation being worse than if you require explicit casting, because then what you're doing and what you have to be careful about are less obvious, since what's going on is hidden.

> The other option would be deep copying or serialization.

That would be far too costly IMHO. In the vast majority of cases (in my experience at least and from what I've seen others do), what you really want to do is pass ownership of the object from one thread to the other, and while deep copying would allow you to avoid type system issues, it's completely unnecessary otherwise. So, we'd be introducing overhead just to satisfy our very restrictive type system. The only way that I can think of to fix that would be for objects to all have a concept of what thread owns them (so that the type system would be able to understand the concept of an object's ownership being passed from one thread to another), but that would be a _big_ change and likely way too complicated in general.

- Jonathan M Davis

On Oct 10, 2013, at 10:43 AM, Joseph Rushton Wakeling <joseph.wakeling@webdrake.net> wrote:

> On 10/10/13 19:39, Sean Kelly wrote:
>> As long as the reference itself can be reassigned (tail-immutable, I suppose) I think immutable is occasionally quite useful for complex data structures.  It basically formalizes the RCU (read-copy-update) approach to wait-free concurrency.  I'd tend to use this most often for global data structures built up on app start, and updated rarely to never as the program runs.
> 
> This kind of stuff is outside my experience, so if you'd like to offer a more detailed explanation/example, I'd be very grateful :-)

Configuration data, for example.  On app start you might load a config file, generate information about the user, and so on, before real processing begins.  This data needs to be visible everywhere and it rarely if ever changes as the program runs, so you fill the data structures and then make them immutable.  Assuming, of course, that the data structures have immutable versions of all the necessary functions (which is unfortunately a pretty big assumption).

On 10/10/13 12:18 AM, Walter Bright wrote:
> On 10/9/2013 11:34 PM, Jacob Carlborg wrote:
>> On 2013-10-10 02:22, Sean Kelly wrote:
>>
>>> Only that this would have to be communicated to the user, since
>>> moving data
>>> later is problematic. Today, I think it's common to construct an
>>> object as
>>> unshared and then cast it.
>>
>> What is the reason to not create it as "shared" in the first place?
>
> 1. Shared data cannot be passed to regular functions.

I don't understand this. If a function/method accepts "shared", then it can be passed shared data.

> 2. Functions that create data structures would have to know in advance
> that they'll be creating a shared object. I'm not so sure this would not
> be an invasive change.

There is no other way around it. And this is not a change - it's fixing something.

> 3. Immutable data is implicitly shared. But it is not created immutable
> - it is created as mutable data, then set to some state, then cast to
> immutable.

That all must happen in the runtime, NOT in user code.


Andrei

On Oct 10, 2013, at 10:50 AM, "Jonathan M Davis" <jmdavisProg@gmx.com> wrote:

> On Thursday, October 10, 2013 10:27:24 Sean Kelly wrote:
>> On Oct 9, 2013, at 9:24 PM, Jonathan M Davis <jmdavisProg@gmx.com> wrote:
>>> And given that std.concurrency requires casting to and from shared or immutable in order to pass objects across threads, it seems ilke most of D's concurrency model requires casting to and/or from shared or immutable.
>> std.concurrency won't be this way forever though. We could fake move semantics with something like assumeUnique!T, so send could be modified to accept a non-shared class that's marked as Unique.
> 
> I take it that you mean something other than std.exception.assumeUnique which simply casts to immutable? All that std.exception.assumeUnique does for you over casting is document why the cast is happening.
> 
> If you're talking about creating a wrapper type which indicates that the object is unique, I'd still expect that the casting would have to be happening underneath the hood when the object was passed (though then for better or worse, it would be encapsulated). And unless the objecte were always in that Unique wrapper, the programmer would still have to be promising that the object was actually unique and not being shared across threads rather than the type system doing it, in which case, I don't see much gain over simply casting. And if it's always in the wrapper, then you're in a similar boat to shared or immutable in that it's not the correct type.
> 
> I expect that there are nuances in what you're suggesting that I don't grasp at the moment, but as far as I can tell, the type system fundamentally requires a cast when passing objects across threads. It's just a question of whether that cast is hidden or not, and depending on how you hide it, I think that there's a real risk of the situation being worse than if you require explicit casting, because then what you're doing and what you have to be careful about are less obvious, since what's going on is hidden.

Yes, we couldn't use assumeUnique as-is because then the object would land on the other side as immutable.  It would have to wrap the object to tell send() that the object, while not shared or immutable, is safe to put in a message.  Then send() would discard the wrapper while constructing the message.

On 10/10/13 12:33 AM, Jonathan M Davis wrote:
> I honestly don't think we can solve it a different way without completely
> redesigning shared. shared is specifically designed such that you have to
> either cast it way to do anything with it

no

> or write all of your code to
> explicitly work with shared, which is not something that generally makes sense
> to do unless you're creating a type whose only value is in being shared across
> threads.

yes

> Far more frequently, you want to share a type which you would also
> use normally as a thread-local variable, and that means casting.

no


Andrei

On Oct 10, 2013, at 10:55 AM, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:

> On 10/10/13 12:33 AM, Jonathan M Davis wrote:
> 
>> Far more frequently, you want to share a type which you would also use normally as a thread-local variable, and that means casting.
> 
> no

Yeah, I'd want to see this claim backed up by some examples.  The only data I share globally in my own apps is the occasional container.  Configuration data, a user database, whatever.  I'll also frequently move data between threads while dispatching tasks, but otherwise everything is thread-local.  I imagine there are other reasonable methods for using shared data, but I don't know what they are.