Graham St Jack wrote:
> Michel Fortin wrote:
>> Le 2010-01-10 ? 22:47, Graham St Jack a ?crit :
>>
>> 
>>> I'm happy with things being carefully wrapped and auto-magic. However, surely the message channel (or whatever it is called) object itself will have be shared (and its externally accessible methods synchronized), otherwise we are cheating. We have a object whose sole purpose in life is to be shared by multiple threads, so surely if anything is shared, this one has to be.
>>>
>>> If this is the case, then we need a mechanism to stop "shared" from leaking out into the whole code base.
>>> 
>>
>> When your browser requests a web page to a server, it sends messages and receives back replies. But it doesn't need the browser need to have a "shared" object with the server. Instead, you have a socket on each side and both get connected through the operating system's TCP/IP stack auto-magic.
>>
>> In the same way, a message-passing API, especially one that intends to go beyond threads, doesn't need and shouldn't expose any shared object. That doesn't mean the implementation won't use shared for thread-to-thread communications, but the user of the API shouldn't need to see that.
>>
>>
>> 
> I understand what you are saying, but your example is for inter-process communications, and the shared objects you talk about are external to the process, in non-D code.
> 
> I also understand that the messaging library is attempting to make inter-thread, inter-process and inter-node communications all look the same. I'm not sure this is a good idea, and I for one would like to have a version of the messaging API that used blatantly shared objects for good old-fashioned inter thread message passing. If such a thing doesn't eventuate, I for one would like to be able to build one without having to resort to back-door tricks.

The short answer is that there won't be an easy way to continue "good" (?) old-fashioned multithreaded programming in D with blatant (e.g. unchecked) sharing. That's a bad habit that we don't need and don't want to support.

Inter-thread and inter-process communication will obey different limitations, but they do have a common subset that we'll expose. That will help programmer in creating location-unaware concurrency.

> With communication between threads within a process, we are looking at using the shared keyword to primarily indicate which objects are potentially shared. I still think it is a cop-out to say "this shared object is safe to use, so lets pretend it isn't shared". Sure, that avoids the problems associated with the viral nature of the shared keyword, but does nothing to help us understand how to get the design of shared right.

I don't know what you mean. There's no pretense that shared objects are actually not shared, but I'm sure I'm missing a point.


Andrei

Le 2010-01-11 ? 18:10, Graham St Jack a ?crit :

> Michel Fortin wrote:
>> When your browser requests a web page to a server, it sends messages and receives back replies. But it doesn't need the browser need to have a "shared" object with the server. Instead, you have a socket on each side and both get connected through the operating system's TCP/IP stack auto-magic.
>> 
>> In the same way, a message-passing API, especially one that intends to go beyond threads, doesn't need and shouldn't expose any shared object. That doesn't mean the implementation won't use shared for thread-to-thread communications, but the user of the API shouldn't need to see that.
> 
> I understand what you are saying, but your example is for inter-process communications, and the shared objects you talk about are external to the process, in non-D code.

It doesn't change a thing. Passing objects by reference should be an optimization, not a requirement. It'd be pretty dumb if it wasn't possible between thread, I agree. But we don't need to make the user aware of it.


> I also understand that the messaging library is attempting to make inter-thread, inter-process and inter-node communications all look the same. I'm not sure this is a good idea, and I for one would like to have a version of the messaging API that used blatantly shared objects for good old-fashioned inter thread message passing. If such a thing doesn't eventuate, I for one would like to be able to build one without having to resort to back-door tricks.

You will be capable of making your own communication mechanisms using shared. If you can't, it'd mean that shared is pretty dysfunctional.


> With communication between threads within a process, we are looking at using the shared keyword to primarily indicate which objects are potentially shared. I still think it is a cop-out to say "this shared object is safe to use, so lets pretend it isn't shared". Sure, that avoids the problems associated with the viral nature of the shared keyword, but does nothing to help us understand how to get the design of shared right.

If you're talking about immutable, it was Walter's decision to keep the type system simpler by having immutable and shared-immutable be the same. And I don't think reopening that debate will lead to anything.

If you're talking about the message passing API... well I'm not the one designing it but if I were the internal implementation for thread-to-thread communication would use shared, but the user-visible parts (the API) wouldn't. For instance, you could have this struct:

	struct MessageDispatcher {
		private shared MessageQueue messageQueue;

		void send(string[] message) {
			synchronized (messageQueue) {
				messageQueue.add(message);
			}
		}

		string receive() {
			synchronized (messageQueue) {
				while (messageQueue.empty)
					messageQueue.wait();
				return messageQueue.get();
			}
		}
	}

And you would use the endpoint like this:

	void main() {
		MessageDispatcher dispatcher = createOneSomehow();
		span(&threadFunc, endpoint); // endpoint passed *by copy*
		endpoint.send("hello");
	}

	void threadFunc(MessageDispatcher dispatcher) {
		string message = dispatcher.receive(); // blocking
		// ... use message
	}

I'm not sure how putting "shared" in the user's code would help anything here. Sure, if I wasn't passing an immutable string it wouldn't work, but that's just a crude example. I'd expect the message passing API to accept anything as a message. If the message is neither immutable nor shared, then the message passing API should make a copy; if it is immutable or shared, then it can and should be passed as-is to the other thread. Does that make sense?

-- 
Michel Fortin
michel.fortin at michelf.com
http://michelf.com/

Andrei Alexandrescu wrote:
>> I understand what you are saying, but your example is for inter-process communications, and the shared objects you talk about are external to the process, in non-D code.
>>
>> I also understand that the messaging library is attempting to make inter-thread, inter-process and inter-node communications all look the same. I'm not sure this is a good idea, and I for one would like to have a version of the messaging API that used blatantly shared objects for good old-fashioned inter thread message passing. If such a thing doesn't eventuate, I for one would like to be able to build one without having to resort to back-door tricks.
>
> The short answer is that there won't be an easy way to continue "good" (?) old-fashioned multithreaded programming in D with blatant (e.g. unchecked) sharing. That's a bad habit that we don't need and don't want to support.
>
> Inter-thread and inter-process communication will obey different limitations, but they do have a common subset that we'll expose. That will help programmer in creating location-unaware concurrency.
>
>> With communication between threads within a process, we are looking at using the shared keyword to primarily indicate which objects are potentially shared. I still think it is a cop-out to say "this shared object is safe to use, so lets pretend it isn't shared". Sure, that avoids the problems associated with the viral nature of the shared keyword, but does nothing to help us understand how to get the design of shared right.
>
> I don't know what you mean. There's no pretense that shared objects are actually not shared, but I'm sure I'm missing a point.
Almost all my multi-threaded code uses a synchronized queue template for inter-thread communication, and I always transfer immutable or by-value data.

It looks like this:

Producer Thread ---(add)---> Queue <---(remove)---- Consumer Thread.

The Queue is shared between the two threads, and all of its public methods are synchronized. The Queue is shared, but the messages are immutable.

My assertion is that Queue (and any other classes like it) should be marked as "shared", so we clearly understand where our threads are in contact, and so the compiler can help us keep to the straight and narrow.

My question is "what are we going to do with the language to stop shared from infecting the entire application", which is what always happened to me in the past whenever I attempted to define Queue to be a shared class.

>
>
> Andrei
> _______________________________________________
> dmd-concurrency mailing list
> dmd-concurrency at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-concurrency

You are describing message passing, which in D is virtually built in - you don't need to define your own Queue. There will be a simple send and receive mechanism that I will describe in the next draft.

Andrei

Graham St Jack wrote:
> Andrei Alexandrescu wrote:
>>> I understand what you are saying, but your example is for inter-process communications, and the shared objects you talk about are external to the process, in non-D code.
>>>
>>> I also understand that the messaging library is attempting to make inter-thread, inter-process and inter-node communications all look the same. I'm not sure this is a good idea, and I for one would like to have a version of the messaging API that used blatantly shared objects for good old-fashioned inter thread message passing. If such a thing doesn't eventuate, I for one would like to be able to build one without having to resort to back-door tricks.
>>
>> The short answer is that there won't be an easy way to continue "good" (?) old-fashioned multithreaded programming in D with blatant (e.g. unchecked) sharing. That's a bad habit that we don't need and don't want to support.
>>
>> Inter-thread and inter-process communication will obey different limitations, but they do have a common subset that we'll expose. That will help programmer in creating location-unaware concurrency.
>>
>>> With communication between threads within a process, we are looking at using the shared keyword to primarily indicate which objects are potentially shared. I still think it is a cop-out to say "this shared object is safe to use, so lets pretend it isn't shared". Sure, that avoids the problems associated with the viral nature of the shared keyword, but does nothing to help us understand how to get the design of shared right.
>>
>> I don't know what you mean. There's no pretense that shared objects are actually not shared, but I'm sure I'm missing a point.
> Almost all my multi-threaded code uses a synchronized queue template for inter-thread communication, and I always transfer immutable or by-value data.
> 
> It looks like this:
> 
> Producer Thread ---(add)---> Queue <---(remove)---- Consumer Thread.
> 
> The Queue is shared between the two threads, and all of its public methods are synchronized. The Queue is shared, but the messages are immutable.
> 
> My assertion is that Queue (and any other classes like it) should be marked as "shared", so we clearly understand where our threads are in contact, and so the compiler can help us keep to the straight and narrow.
> 
> My question is "what are we going to do with the language to stop shared from infecting the entire application", which is what always happened to me in the past whenever I attempted to define Queue to be a shared class.
> 
>>
>>
>> Andrei
>> _______________________________________________
>> dmd-concurrency mailing list
>> dmd-concurrency at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/dmd-concurrency
> 
> _______________________________________________
> dmd-concurrency mailing list
> dmd-concurrency at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-concurrency

An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/dmd-concurrency/attachments/20100112/4b0cb634/attachment.htm>

An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/dmd-concurrency/attachments/20100111/8c095aea/attachment.htm>