Hi all, I'm a bit confused today (as usual, haha).

I have a pointer to a struct (let's call it Foo) allocated via a C library.
I need to do some expensive computation with the Foo* to create a Bar[], but I would like to do that computation in the background, because the Bar[] is not needed right away.

I definitely do not want there to be a copy of all elements of the Bar[] between threads, because it is very large.

I tried to implement it like this:

    void fooPtrToBarArr(in shared Foo* f, out shared Bar[] b){ /*do work*/ }
    __gshared Foo* foo;
    foo = allocateFoo();
    __gshared Bar[] bar;
    spawn(foo, bar);

To my dismay, it results in a cryptic compiler error:

> template std.concurrency.spawn cannot deduce function from argument types
> !()(void function(shared(const(Foo*)) f, out shared(Bar[]) b), Foo*,
> Bar[]), candidates are:
> /usr/include/dlang/dmd/std/concurrency.d(466):
> std.concurrency.spawn(F, T...)(F fn, T args) if (isSpawnable!(F, T))

Any help would be greatly appreciated :)

On 08/04/2015 01:03 AM, "岩倉 澪" wrote:
> Hi all, I'm a bit confused today (as usual, haha).
>
> I have a pointer to a struct (let's call it Foo) allocated via a C library.
> I need to do some expensive computation with the Foo* to create a Bar[],
> but I would like to do that computation in the background, because the
> Bar[] is not needed right away.
>
> I definitely do not want there to be a copy of all elements of the Bar[]
> between threads, because it is very large.
>
> I tried to implement it like this:
>
>      void fooPtrToBarArr(in shared Foo* f, out shared Bar[] b){ /*do
> work*/ }
>      __gshared Foo* foo;
>      foo = allocateFoo();
>      __gshared Bar[] bar;
>      spawn(foo, bar);
>
> To my dismay, it results in a cryptic compiler error:
>
>> template std.concurrency.spawn cannot deduce function from argument types
>> !()(void function(shared(const(Foo*)) f, out shared(Bar[]) b), Foo*,
>> Bar[]), candidates are:
>> /usr/include/dlang/dmd/std/concurrency.d(466):
>> std.concurrency.spawn(F, T...)(F fn, T args) if (isSpawnable!(F, T))
>
> Any help would be greatly appreciated :)
>
>

__gshared behaves like C globals and need not be passed to spawned functions. (Although, if needed, they must be passed as shared and casted back to non-shared in the thread function.)

The following seems to achieve what you describe:

import std.stdio;
import std.concurrency;
import core.thread;

struct Foo
{}

struct Bar
{
    int i;
}

void fooPtrToBarArr()
{
    bar ~= Bar(42);
}

__gshared Foo* foo;
__gshared Bar[] bar;

void main()
{
    spawn(&fooPtrToBarArr);
    thread_joinAll();

    writeln(bar);
}

Ali

import std.concurrency;
import std.typecons : Unique;
import std.exception : assumeUnique;

struct Foo { }

struct Bar { }

void bar_generator (Tid ownerTid)
{
    receive(
        (shared(Foo)* input) {
            auto output = new Bar[100];
            // compute output ..
            // .. and cast to immutable via assumeUnique when
            // finished as it won't be mutated anymore and no
            // other pointers exist
            send(ownerTid, assumeUnique(output));
        }
    );
}

void main ()
{
    auto generator = spawn(&bar_generator, thisTid);

    // must be shared or immutable to be passed between threads
    auto input = new shared Foo;
    send(generator, input);

    // in real app use `receiveTimeout` to do useful stuff until
    // result message is received
    auto output = receiveOnly!(immutable(Bar)[]);

    import std.stdio;
    writeln(output.length);
}

On Tuesday, 4 August 2015 at 08:03:54 UTC, 岩倉 澪 wrote:
> Hi all, I'm a bit confused today (as usual, haha).
>
> I have a pointer to a struct (let's call it Foo) allocated via a C library.
> I need to do some expensive computation with the Foo* to create a Bar[], but I would like to do that computation in the background, because the Bar[] is not needed right away.
>
> I definitely do not want there to be a copy of all elements of the Bar[] between threads, because it is very large.
>
> I tried to implement it like this:
>
>     void fooPtrToBarArr(in shared Foo* f, out shared Bar[] b){ /*do work*/ }
>     __gshared Foo* foo;
>     foo = allocateFoo();
>     __gshared Bar[] bar;
>     spawn(foo, bar);
>
> To my dismay, it results in a cryptic compiler error:
>
>> template std.concurrency.spawn cannot deduce function from argument types
>> !()(void function(shared(const(Foo*)) f, out shared(Bar[]) b), Foo*,
>> Bar[]), candidates are:
>> /usr/include/dlang/dmd/std/concurrency.d(466):
>> std.concurrency.spawn(F, T...)(F fn, T args) if (isSpawnable!(F, T))
>
> Any help would be greatly appreciated :)

Do you mean this instead?

spawn(&fooPtrToBarArr, foo, bar);

Anyway, you need to use shared, not __gshared, then it should work. E.g.

import std.concurrency;

struct Foo{}
auto allocateFoo() { return new Foo(); }
struct Bar{}

void fooPtrToBarArr(in shared Foo* f, out shared Bar[] b){
    /*do work*/ }

void main()
{
    shared Foo* foo = cast(shared)allocateFoo();
    shared Bar[] bar;

    spawn(&fooPtrToBarArr, foo, bar);
}

Then you will have to cast away shared to make use of bar in your normal code (or continue using it as shared, but that would be frustrating). To totally avoid any ordering concerns, you could put a full mfence in before casting away shared (http://dlang.org/phobos/core_atomic.html#.atomicFence).

On Tuesday, 4 August 2015 at 08:35:10 UTC, Dicebot wrote:
>     auto output = receiveOnly!(immutable(Bar)[]);

Won't message passing like this result in an expensive copy, or does the cast to immutable via assumeUnique avoid that?

On Tuesday, 4 August 2015 at 08:36:26 UTC, John Colvin wrote:
> Do you mean this instead?
>
> spawn(&fooPtrToBarArr, foo, bar);

Yep, that was a typo when writing up the post!

> Anyway, you need to use shared, not __gshared, then it should work.

I have been wary of shared because of: https://p0nce.github.io/d-idioms/#The-truth-about-shared

On Tue, 04 Aug 2015 10:29:55 +0000
"岩倉 澪" <mio.iwakura@gmail.com> wrote:

> On Tuesday, 4 August 2015 at 08:35:10 UTC, Dicebot wrote:
> >     auto output = receiveOnly!(immutable(Bar)[]);
> 
> Won't message passing like this result in an expensive copy

No it will copy only struct containing length and pointer to data, so it is cheap.

On Tuesday, 4 August 2015 at 10:29:57 UTC, 岩倉 澪 wrote:
> On Tuesday, 4 August 2015 at 08:35:10 UTC, Dicebot wrote:
>>     auto output = receiveOnly!(immutable(Bar)[]);
>
> Won't message passing like this result in an expensive copy, or does the cast to immutable via assumeUnique avoid that?

immutable data is implicitly shared and doesn't need to be duplicated - as it won't ever be modified, it is perfectly fine to do concurrent reads for that from multiple references.

std.concurrency does by-value message passing (in this case just ptr+length), it never deep copies automatically

On Tuesday, 4 August 2015 at 10:37:39 UTC, Dicebot wrote:
> std.concurrency does by-value message passing (in this case just ptr+length), it never deep copies automatically

I assumed that it would deep copy (in the case of mutable data) since the data being sent is thread-local (unless I am misunderstanding something)

On Tuesday, 4 August 2015 at 11:33:11 UTC, 岩倉 澪 wrote:
> On Tuesday, 4 August 2015 at 10:37:39 UTC, Dicebot wrote:
>> std.concurrency does by-value message passing (in this case just ptr+length), it never deep copies automatically
>
> I assumed that it would deep copy (in the case of mutable data) since the data being sent is thread-local (unless I am misunderstanding something)

It is heap-allocated and in there is no thread-local heap currently in D - only globals and static variables.

std.concurrency never deep copies - if you are trying to send data which contains indirections (pointers/arrays) _and_ is not marked either immutable or shared, it will simply not compile.