My task involves many worker threads, each of them uses Curl instance to do one or several requests and one Mysql connection instance to store data and do some aggregations. ALso sometimes these threads do other work that don't require these resources.

So I want to make two pools (one for Curl and one for Mysql connections), so that number of DB connections (or Curl instances) will be lower than number of threads, so I cannot just use DataPool.WorkerLocalStorageRange.

So I'm trying to create shared pool. And there arises question:

if I have this code:


class SharedPool {

...
	Mysql* get() {
...
		return cast(Mysql*) connections[freeIndex];
	}
...
	Mysql*[] connections;
}

shared SharedPool pool;


when I will get syncronization overhead: only on access to SharedPool or on every access to Mysql instance?

What I really want - is to NOT have any shared-related code after getting some Mysql instance and before returning it to pool, so Mysql instances must be essentially non-shared. Is it possible?

Am 17.03.2013 22:21, schrieb Alex Khmara:
> My task involves many worker threads, each of them uses Curl instance to
> do one or several requests and one Mysql connection instance to store
> data and do some aggregations. ALso sometimes these threads do other work
> that don't require these resources.
>
> So I want to make two pools (one for Curl and one for Mysql connections),
> so that number of DB connections (or Curl instances) will be lower than
> number of threads, so I cannot just use DataPool.WorkerLocalStorageRange.
>
> So I'm trying to create shared pool. And there arises question:
>
> if I have this code:
>
>
> class SharedPool {
>
> ...
> 	Mysql* get() {
> ...
> 		return cast(Mysql*) connections[freeIndex];
> 	}
> ...
> 	Mysql*[] connections;
> }
>
> shared SharedPool pool;
>
>
> when I will get syncronization overhead: only on access to SharedPool or
> on every access to Mysql instance?
>
> What I really want - is to NOT have any shared-related code after getting
> some Mysql instance and before returning it to pool, so Mysql instances
> must be essentially non-shared. Is it possible?
>

Your current code will not work at all, because you can not call get() from shared instance of the class (because get is not a shared method). Also your current code does not have any synchroization overhead. Just adding shared to something does not mean that there will be any synchronization added automatically (at least not yet). You need to add synchronization yourself for example by using a "synchronized(this) { ... }" block.

Kind Regards
Benjamin Thaut

It seems that I was little bit inaccurate in my question.

More correctly, code is this (int template param is just a hack, I plan to do more correct version later):

//************************************

class Pool(T, U = int) {
	this() {
		mutex = new Mutex;
	}
	struct PoolRef(T) {
		T* member;
		Pool!T *pool;
		alias member this;
		~this() {
			pool.free(member);
		}
	};

	synchronized PoolRef!T get() {
		if (!mutex) {
			init();
		}
		if (firstFree >= items.length) {
			if (items.length < maxMembers) {
				items ~= T();
				static if (!is(U == int)) {
					initLambda(items[$]);
				}
				free ~= true;
			} else {
				throw new Exception("Too many requests to  pool");
			}
		}
		free[firstFree] = false;
		for (; firstFree < items.length && !free[firstFree]; ++firstFree)
			{}
		PoolRef pr;
		pr.member = items[$];
		pr.pool = this;
		return pr;
	}

	private:
		U initLambda;
		class Mutex {};
		T*[] items;
		bool[] free;
		int firstFree;
		int maxMembers;
		shared Mutex mutex;
}

shared Pool!Mysql pool;

//************************************

This compiles (and simplified example wit big sleeps in get and debug output seems to work correctly), but I am not sure about thread-safety.

If after that I will have no no shared-related overhead in call to Mysql methods than things are good - pool will not give Mysql instance to second thread until it will be relased by first thread.

On Mon, 18 Mar 2013 08:28:41 +0100, Benjamin Thaut wrote:

> Am 17.03.2013 22:21, schrieb Alex Khmara:
>> My task involves many worker threads, each of them uses Curl instance to do one or several requests and one Mysql connection instance to store data and do some aggregations. ALso sometimes these threads do other work that don't require these resources.
>>
>> So I want to make two pools (one for Curl and one for Mysql
>> connections),
>> so that number of DB connections (or Curl instances) will be lower than
>> number of threads, so I cannot just use
>> DataPool.WorkerLocalStorageRange.
>>
>> So I'm trying to create shared pool. And there arises question:
>>
>> if I have this code:
>>
>>
>> class SharedPool {
>>
>> ...
>> 	Mysql* get() {
>> ...
>> 		return cast(Mysql*) connections[freeIndex];
>> 	}
>> ...
>> 	Mysql*[] connections;
>> }
>>
>> shared SharedPool pool;
>>
>>
>> when I will get syncronization overhead: only on access to SharedPool or on every access to Mysql instance?
>>
>> What I really want - is to NOT have any shared-related code after getting some Mysql instance and before returning it to pool, so Mysql instances must be essentially non-shared. Is it possible?
>>
>>
> Your current code will not work at all, because you can not call get() from shared instance of the class (because get is not a shared method). Also your current code does not have any synchroization overhead. Just adding shared to something does not mean that there will be any synchronization added automatically (at least not yet). You need to add synchronization yourself for example by using a "synchronized(this) { ... }" block.
> 
> Kind Regards Benjamin Thaut

Am 18.03.2013 20:55, schrieb Alex Khmara:
> This compiles (and simplified example wit big sleeps in get and debug output seems to work correctly),
> but I am not sure about thread-safety.

Looks thread safe for me.

>
> If after that I will have no no shared-related overhead in call to Mysql methods than
> things are good - pool will not give Mysql instance to second thread until it will be relased by first thread.

The only problem you may run into, is that the destructor of PoolRef might gets called multiple times if you are not really carefull with temporary variables. Ideal would be a way to disallow copying of the struct and only allow moving it, last I checked this was not possible in D. Because when you do @disable this(this) it will also disallow moving the struct. Although you could try with the current version.

Kind Regards
Benjamin Thaut

Thanks, Benjamin.

I'll try to somehow disable copying. At the end I can change struct to
scoped class and write custom opAssign for it...
Well, I need to play with it.

Thanks you again.

On Mon, 18 Mar 2013 22:05:51 +0100, Benjamin Thaut wrote:

> Am 18.03.2013 20:55, schrieb Alex Khmara:
>> This compiles (and simplified example wit big sleeps in get and debug
output seems to work correctly),
>> but I am not sure about thread-safety.
> 
> Looks thread safe for me.
> 
>>
>> If after that I will have no no shared-related overhead in call to
Mysql methods than
>> things are good - pool will not give Mysql instance to second thread
until it will be relased by first thread.
> 
> The only problem you may run into, is that the destructor of PoolRef might gets called multiple times if you are not really carefull with temporary variables. Ideal would be a way to disallow copying of the struct and only allow moving it, last I checked this was not possible
in
> D. Because when you do @disable this(this) it will also disallow moving the struct. Although you could try with the current version.
> 
> Kind Regards
> Benjamin Thaut

Hmm, it seems that std.typecons. Unique does exactly move semantics. It accepts pointer, so I get another indirection level, but for my purposes it's fine.

On Mon, 18 Mar 2013 22:05:51 +0100, Benjamin Thaut wrote:

> 
> The only problem you may run into, is that the destructor of PoolRef might gets called multiple times if you are not really carefull with temporary variables. Ideal would be a way to disallow copying of the struct and only allow moving it, last I checked this was not possible in D. Because when you do @disable this(this) it will also disallow moving the struct. Although you could try with the current version.