I have the folowing problem:
I like to envelope the class object in struct to control the destruction moment and then send this object to another thread/fiber (or task, cause I use vibe-d).

I can't find any method to make it working. Any ideas?

dmd (version 075) gives so stupid results, I belive it's broken (I've created the issue https://issues.dlang.org/show_bug.cgi?id=17749 )

ldc2 returns some errors:

.../import/std/variant.d(610,27): Error: function core.stdc.string.memcpy (void* s1, const(void*) s2, ulong n) is not callable using argument types (ubyte[32]*, shared(Env!(shared(A)))*, ulong)
.../import/std/conv.d(4186,9): Error: static assert  "Cannot emplace a Env!(shared(A)) because Env!(shared(A)).this(this) is annotated with @disable."
.../import/std/conv.d(4198,24):        instantiated from here: emplaceRef!(Env!(shared(A)), Env!(shared(A)), Env!(shared(A)))
.../import/std/variant.d(301,35):        instantiated from here: emplaceRef!(Env!(shared(A)), Env!(shared(A)))
.../import/std/variant.d(630,21):        instantiated from here: handler!(shared(Env!(shared(A))))
.../import/std/variant.d(544,17):        ... (5 instantiations, -v to show) ...
../../../.dub/packages/vibe-core-1.1.1/vibe-core/source/vibe/core/concurrency.d(1223,64):        instantiated from here: send!(shared(Env!(shared(A))))
app.d(74,7):        instantiated from here: send!(shared(Env!(shared(A))))



The code may be complied like this: dub --single --compiler=ldc2 ./app.d   (assuming it's saved in app.d file).


/+
dub.sdl:
name "simple"
dependency "vibe-core" version="~>1.1.0"
+/
import std.stdio;
import std.format;

import vibe.core.core;
import vibe.core.task;
import vibe.core.concurrency;

// simple class with destructor
shared class A
{
	int i;
	this(int i)
	{
		this.i = i;
	}

	~this()
	{
		writeln("destruct ", i);
	}

}

shared struct Env(T) if (is(T == class) && is(T == shared))
{
	T obj;

	this(T o)
	{
		obj = obj;
	}

	this(this)
	{
		// some magic here
	}

	auto opAssign(shared(Env!T) other)
	{
		obj = other.obj;
		return this;
	}

	auto opAssign(ref shared(Env!T) other)
	{
		obj = other.obj;
		return this;
	}

	~this()
	{
		if (obj !is null)
			destroy(obj);
		obj = null;
	}
}

void main()
{
	auto consumer = runTask({
		auto got = receiveOnly!(shared Env!(shared A))();
		writeln(typeof(got).stringof);
	});

	auto producer = runTask({
		auto a = new shared A(1);
		shared b = shared Env!(shared A)(a);
		writeln(typeof(b).stringof);
		send(consumer, b);
	});
	runApplication();
}

On Saturday, 12 August 2017 at 18:57:44 UTC, Arek wrote:
> I have the folowing problem:
> I like to envelope the class object in struct to control the destruction moment and then send this object to another thread/fiber (or task, cause I use vibe-d).
>
> I can't find any method to make it working. Any ideas?

I tried it too some time ago. Then I read Alexandrescu book and realized that the authors of the language do not want anyone to do this. Long story short, just plan your application so that each complex object is monitored by only one thread/task, and pass not objects, but messages (immutable structs) about what to do with them.

On Sunday, 13 August 2017 at 02:50:13 UTC, crimaniak wrote:
> On Saturday, 12 August 2017 at 18:57:44 UTC, Arek wrote:
>> I have the folowing problem:
>> I like to envelope the class object in struct to control the destruction moment and then send this object to another thread/fiber (or task, cause I use vibe-d).
>>
>> I can't find any method to make it working. Any ideas?
>
> I tried it too some time ago. Then I read Alexandrescu book and realized that the authors of the language do not want anyone to do this. Long story short, just plan your application so that each complex object is monitored by only one thread/task, and pass not objects, but messages (immutable structs) about what to do with them.

Yeah, I've read this. But conurrency.send cannot pass immutable object. The same story with Unique.
What means, that if I have complex objects which I want to transfer between threads I must keep them somewhere in global memory and share among threads. I can use send/receive only to signal availability of their existance. What I need is kind of "move" operation beetwen threads.

More over, "shared" looks rather like unfinished concept. It is really difficult to create proper struct (with postblit and destructor) working as shared object.

I even have no clue what is exact semantics of "shared" types. Language specification is a little laconic.

Anyway, _gshared looks very promising, so I will try to work out any approach.

Thanks for answer.
Arek

On Sunday, 13 August 2017 at 11:35:05 UTC, Arek wrote:
> Yeah, I've read this. But conurrency.send cannot pass immutable object. The same story with Unique.
 Sorry, read this as 'efficient immutable'.

> More over, "shared" looks rather like unfinished concept.
 Yes, exactly.

> Anyway, _gshared looks very promising, so I will try to work out any approach.
More of this, I think, you can't avoid __gshared for any complex work. Even mutexes from Phobos doesn't support shared, so I had to 'cowboy with __gshared' when implementing my site engine.

On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn wrote:
> I have the folowing problem:
> I like to envelope the class object in struct to control the
> destruction moment and then send this object to another
> thread/fiber (or task, cause I use vibe-d).
>
> I can't find any method to make it working. Any ideas?

Unfortunately, send and receive do not currently work with shared because of issues with Variant, which they use internally.

- Jonathan M Davis

On Sunday, August 13, 2017 16:40:03 crimaniak via Digitalmars-d-learn wrote:
> More of this, I think, you can't avoid __gshared for any complex work. Even mutexes from Phobos doesn't support shared, so I had to 'cowboy with __gshared' when implementing my site engine.

The way to handle shared is to protect the section of code that's using the shared object with either a mutex or synchronized block, and then you cast away shared from the object within that section and operate on it as thread-local. When you're done, you make sure that you don't have any thread-local references to the data, and you release the mutex or exit the synchronized block. e.g. something like

shared T sharedObj = getSharedObj();

synchronized(mutex)
{
    T nonSharedObj = cast(T)sharedObject

    // do stuff...

    // make sure that no references to nonSharedObj have escaped
}

// now, there's just the shared version of the object

And no, this isn't ideal, but the only semi-decent solution that's been proposed that safely casts away shared for you is synchronized classes, which Andrei describes in TDPL but have never been implemented. And because they can only safely strip off the outermost layer of shared, they're of questionable usefulness anyway. Ultimately, even with synchronized classes, in many situations, the programmer is going to have to carefully cast away shared to operate on the object within a protected context.

Now, the fact that the mutex objects don't handle shared correctly is another issue entirely. Having to cast away shared from mutexes is dumb, because you're obviously not going to be protecting them with a mutex, and their operations have to be atomic anyway for them to do what they do. So, that definitely needs to be fixed. However, I believe that it _has_ been fixed in master, and it might have made it into a release now, but I'm not sure. So, core.sync.mutex.Mutex _should_ now be useable as shared like it should be.

In general though, the idea is that you simply don't operate on shared objects except via atomic operations. Otherwise, you risk concurrency problems. And really, this is the same as what you'd do in C/C++, except that in C/C++, it doesn't catch you when you operate on an object that's shared across threads with non-atomic operations (because the object isn't explicitly typed as shared), and you don't have to cast away shared to do non-atomic operations. So, having to cast away shared is the price of getting the protection against accidentally using non-atomic operations on a shared object as well as the price we pay to be able to have the type system distinguish between shared and thread-local objects so that it's able to optimize based on the knowledge that an object is thread-local. Ultimately though, you're doing the same thing that you'd do in C++ if you're handling concurrency safely. You just have to explicitly mark stuff as shared and carefully cast away shared in certain, protected contexts.

Using __gshared in extern(D) code is just asking for it, because then you have an object that the compiler thinks is thread-local but isn't, and you risk subtle and nasty bugs as a result. __gshared is only intended for binding to extern(C), global variables. To an extent, you can get away with using it with extern(D) variables, but that's not its intended purpose, and you risk running afoul of the compiler and what it chooses to do based on the assumption that the object is thread-local.

- Jonathan M Davis

On Monday, 14 August 2017 at 03:59:48 UTC, Jonathan M Davis wrote:
> [snip]
>
> Now, the fact that the mutex objects don't handle shared correctly is another issue entirely. Having to cast away shared from mutexes is dumb, because you're obviously not going to be protecting them with a mutex, and their operations have to be atomic anyway for them to do what they do. So, that definitely needs to be fixed. However, I believe that it _has_ been fixed in master, and it might have made it into a release now, but I'm not sure. So, core.sync.mutex.Mutex _should_ now be useable as shared like it should be.
>

My fixes for shared(Mutex) - https://github.com/dlang/druntime/pull/1728 - are part of the DMD v2.074.0 release, and should propagate to LDC with their 1.4 release respectively. GDC master now at druntime/phobos 2.074.1 - https://github.com/D-Programming-GDC/GDC/pull/539 and they're in the process of moving to 2.075.0 - https://github.com/D-Programming-GDC/GDC/pull/542.

Unfortunately, I forgot to write a changelog entry when I was working on the PR and later didn't have time to do so, before the 2.074.0 release.

So shared (Mutex) should be in good shape now. I should supplement the ddoc unittest to address the transitive you issues you talked about. Currently it only shows synchronized use of a plain value type without indirections, which is uninteresting.
The example I'm thinking about is casting with HeadUnsharedOf / TailSharedOf a pointer to a singly-linked list, because with pointers you could show the shared(Node*) -> shared(Node)* transition in the type system, which unfortunately can't be demonstrated with implicitly reference types (classes).

Speaking of TailSharedOf, Jonathan could take a look at this PR: https://github.com/dlang/druntime/pull/1605? I and Andrei already approved it, but I'd like to get a third opinion since this is a breaking change and Martin seems be too busy the moment.

On Monday, 14 August 2017 at 03:40:26 UTC, Jonathan M Davis wrote:
> On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn wrote:
>> I have the folowing problem:
>> I like to envelope the class object in struct to control the
>> destruction moment and then send this object to another
>> thread/fiber (or task, cause I use vibe-d).
>>
>> I can't find any method to make it working. Any ideas?
>
> Unfortunately, send and receive do not currently work with shared because of issues with Variant, which they use internally.
>
> - Jonathan M Davis

That's what I suspected. Thanks for the confirmation. I was not sure if I was doing everything right.

Arek

On 8/13/17 11:40 PM, Jonathan M Davis via Digitalmars-d-learn wrote:
> On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn wrote:
>> I have the folowing problem:
>> I like to envelope the class object in struct to control the
>> destruction moment and then send this object to another
>> thread/fiber (or task, cause I use vibe-d).
>>
>> I can't find any method to make it working. Any ideas?
> 
> Unfortunately, send and receive do not currently work with shared because of
> issues with Variant, which they use internally.

This can't be a correct statement. This is the whole point of shared.

-Steve

On Monday, 14 August 2017 at 03:59:48 UTC, Jonathan M Davis wrote:
> On Sunday, August 13, 2017 16:40:03 crimaniak via Digitalmars-d-learn wrote:
>> More of this, I think, you can't avoid __gshared for any complex work. Even mutexes from Phobos doesn't support shared, so I had to 'cowboy with __gshared' when implementing my site engine.
>
> The way to handle shared is to protect the section of code that's using the shared object with either a mutex or synchronized block, and then you cast away shared from the object within that section and operate on it as thread-local. When you're done, you make sure that you don't have any thread-local references to the data, and you release the mutex or exit the synchronized block. e.g. something like
>
> shared T sharedObj = getSharedObj();
>
> synchronized(mutex)
> {
>     T nonSharedObj = cast(T)sharedObject
>
>     // do stuff...
>
>     // make sure that no references to nonSharedObj have escaped
> }
>
> // now, there's just the shared version of the object
>

Yeah, and this is what i'm  doing now (more or less).
To be more precise, I don't even want to synchronize access to the shared resource between the threads. I just want to move the object from one thread to another.
Of course I could copy the local object, but my obcjecs has indirect references to others, forming kind of tree.

I like the idea of channels in Go. I've tried to get something similiar with send/receive.
In this case, object could be also immutable, because once there are created (in the deserialization process) they will no be modified. I just have to emit them into another task (to be honest, I use fiber, so it's not even another thread, and they will not be accessed in parallel).

But all this language protections makes the issue unexpectedly complicated.



> And no, this isn't ideal, but the only semi-decent solution that's been proposed that safely casts away shared for you is synchronized classes, which Andrei describes in TDPL but have never been implemented. And because they can only safely strip off the outermost layer of shared, they're of questionable usefulness anyway. Ultimately, even with synchronized classes, in many situations, the programmer is going to have to carefully cast away shared to operate on the object within a protected context.
>
> Now, the fact that the mutex objects don't handle shared correctly is another issue entirely. Having to cast away shared from mutexes is dumb, because you're obviously not going to be protecting them with a mutex, and their operations have to be atomic anyway for them to do what they do. So, that definitely needs to be fixed. However, I believe that it _has_ been fixed in master, and it might have made it into a release now, but I'm not sure. So, core.sync.mutex.Mutex _should_ now be useable as shared like it should be.
>
> In general though, the idea is that you simply don't operate on shared objects except via atomic operations. Otherwise, you risk concurrency problems. And really, this is the same as what you'd do in C/C++, except that in C/C++, it doesn't catch you when you operate on an object that's shared across threads with non-atomic operations (because the object isn't explicitly typed as shared), and you don't have to cast away shared to do non-atomic operations. So, having to cast away shared is the price of getting the protection against accidentally using non-atomic operations on a shared object as well as the price we pay to be able to have the type system distinguish between shared and thread-local objects so that it's able to optimize based on the knowledge that an object is thread-local. Ultimately though, you're doing the same thing that you'd do in C++ if you're handling concurrency safely. You just have to explicitly mark stuff as shared and carefully cast away shared in certain, protected contexts.
>
> Using __gshared in extern(D) code is just asking for it, because then you have an object that the compiler thinks is thread-local but isn't, and you risk subtle and nasty bugs as a result. __gshared is only intended for binding to extern(C), global variables. To an extent, you can get away with using it with extern(D) variables, but that's not its intended purpose, and you risk running afoul of the compiler and what it chooses to do based on the assumption that the object is thread-local.
>
> - Jonathan M Davis

Thanks for the explanation! It would be good to have a comprehensive article on this subject.

Arek