Jump to page: 1 2
Thread overview
How to work around the infamous dual-context when using delegates together with std.parallelism
May 27, 2021
Christian Köstlin
May 27, 2021
sighoya
May 27, 2021
Christian Köstlin
May 27, 2021
sighoya
May 27, 2021
Christian Köstlin
May 27, 2021
sighoya
May 27, 2021
Christian Köstlin
May 31, 2021
CandG
May 31, 2021
Christian Köstlin
May 31, 2021
Christian Köstlin
May 27, 2021
Christian Köstlin
May 27, 2021
Ali Çehreli
May 27, 2021
Ali Çehreli
May 27, 2021
Christian Köstlin
May 27, 2021
I have this small program here

test.d:
```
import std;
string doSomething(string[] servers, string user) {
    return user ~ servers[0];
}
void main() {
    auto servers = ["s1", "s2", "s3"];
    auto users = ["u1", "u2", "u3"];
    writeln(map!(user => servers.doSomething(user))(users));
    writeln(taskPool.amap!(user => servers.doSomething(user))(users));
}
```

The first map just works as expected, for the parallel amap though fromo (https://dlang.org/phobos/std_parallelism.html) I get the following warning with dmd:

```
/Users/.../dlang/dmd-2.096.1/osx/bin/../../src/phobos/std/parallelism.d(1711): Deprecation: function `test.main.amap!(string[]).amap` function requires a dual-context, which is deprecated
```

for ldc the build fails with:
```
/Users/.../dlang/ldc-1.26.0/bin/../import/std/parallelism.d(1711): Deprecation: function `test.main.amap!(string[]).amap` function requires a dual-context, which is deprecated
test.d(9):        instantiated from here: `amap!(string[])`
/Users/.../dlang/ldc-1.26.0/bin/../import/std/parallelism.d(1711): Error: function `test.main.amap!(string[]).amap` requires a dual-context, which is not yet supported by LDC
```


Thanks in advance for you insights,
Christian
May 27, 2021

On Thursday, 27 May 2021 at 09:58:40 UTC, Christian Köstlin wrote:

>

I have this small program here

test.d:

import std;
string doSomething(string[] servers, string user) {
    return user ~ servers[0];
}
void main() {
    auto servers = ["s1", "s2", "s3"];
    auto users = ["u1", "u2", "u3"];
    writeln(map!(user => servers.doSomething(user))(users));
    writeln(taskPool.amap!(user => servers.doSomething(user))(users));
}

I think it relates to https://issues.dlang.org/show_bug.cgi?id=5710

The reason is that amap requires a this pointer of type TaskPool and a context pointer to the closure which belongs to main, at least because it requires servers. Having both isn't possible due to problems in non DMD compilers.

If you rewrite it more statically:

string doSomething(string[] servers, string user) {
    return user ~ servers[0];
}

string closure(string user)
{
    return servers.doSomething(user);
}
auto servers = ["s1", "s2", "s3"];
int main()
{
    auto users = ["u1", "u2", "u3"];
    writeln(map!(user => servers.doSomething(user))(users));
    writeln(taskPool.amap!(closure)(users));
    return 0;
}

PS: Just enable markdown if you want to highlight D code

May 27, 2021
Thanks for the proposed solution. It also works in my slightly bigger program (although I do not like to make servers more global).

I tried also the following (which unfortunately also does not work as intended):

```D
import std;
string doSomething(string[] servers, string user) {
    return user ~ servers[0];
}

int main()
{
    auto users = ["u1", "u2", "u3"];
    auto servers = ["s1", "s2", "s3"];
    auto usersWithServers = users.map!(user => tuple!("user", "servers")(user, servers)).array;
    writeln(map!(userWithServers => userWithServers.servers.doSomething(userWithServers.user))(usersWithServers));
    writeln(taskPool.amap!(userWithServers => userWithServers.servers.doSomething(userWithServers.user))(usersWithServers));
    return 0;
}
```

Here I try to put the data I need together into one tuple ("manually") and then pass it all to amap. Can you explain me, where here a double context is needed? Because all data now should be passed as arguments to amap?

Kind regards,
Christian
May 27, 2021
On 2021-05-27 13:11, sighoya wrote:
> On Thursday, 27 May 2021 at 09:58:40 UTC, Christian Köstlin wrote:
>> I have this small program here
>>
>> test.d:
>> ```
>> import std;
>> string doSomething(string[] servers, string user) {
>>     return user ~ servers[0];
>> }
>> void main() {
>>     auto servers = ["s1", "s2", "s3"];
>>     auto users = ["u1", "u2", "u3"];
>>     writeln(map!(user => servers.doSomething(user))(users));
>>     writeln(taskPool.amap!(user => servers.doSomething(user))(users));
>> }
>> ```
> 
> 
> 
> I think it relates to https://issues.dlang.org/show_bug.cgi?id=5710
> 
> The reason is that amap requires a this pointer of type TaskPool and a context pointer to the closure which belongs to main, at least because it requires servers. Having both isn't possible due to problems in non DMD compilers.
> 
> If you rewrite it more statically:
> ```D
> string doSomething(string[] servers, string user) {
>      return user ~ servers[0];
> }
> 
> string closure(string user)
> {
>      return servers.doSomething(user);
> }
> auto servers = ["s1", "s2", "s3"];
> int main()
> {
>      auto users = ["u1", "u2", "u3"];
>      writeln(map!(user => servers.doSomething(user))(users));
>      writeln(taskPool.amap!(closure)(users));
>      return 0;
> }
> ```
> 
> PS: Just enable markdown if you want to highlight D code
On a second not I needed to make server __gshared in my real program, as otherwise its a thread local variable (in the small demo program, this did not occur, I guess because the parallel operations we're too fast).

Kind regards,
Christian
May 27, 2021

On Thursday, 27 May 2021 at 12:17:36 UTC, Christian Köstlin wrote:

>

Can you explain me, where here a double context is needed? Because all data now should be passed as arguments to amap?

Kind regards,
Christian

I believe D's type system isn't smart enough to see independence between context and closure, otherwise your original example would also work as users and servers are context independent.

What about:

string doSomething(string[] servers, string user) {
    return user ~ servers[0];
}
void main() {
    static servers = ["s1", "s2", "s3"];
    static users = ["u1", "u2", "u3"];
    static lambda = (string user) => servers.doSomething(user);
    writeln(map!(user => servers.doSomething(user))(users));
    writeln(taskPool.amap!(lambda)(users));
}
May 27, 2021
On 2021-05-27 14:48, sighoya wrote:
> On Thursday, 27 May 2021 at 12:17:36 UTC, Christian Köstlin wrote:
>> Can you explain me, where here a double context is needed? Because all data now should be passed as arguments to amap?
>>
>> Kind regards,
>> Christian
> 
> I  believe D's type system isn't smart enough to see independence between context and closure, otherwise your original example would also work as users and servers are context independent.
> 
> What about:
> 
> ```D
> string doSomething(string[] servers, string user) {
>      return user ~ servers[0];
> }
> void main() {
>      static servers = ["s1", "s2", "s3"];
>      static users = ["u1", "u2", "u3"];
>      static lambda = (string user) => servers.doSomething(user);
>      writeln(map!(user => servers.doSomething(user))(users));
>      writeln(taskPool.amap!(lambda)(users));
> }
> ```
> 
That looks nice, but unfortunately my data for servers and users in the real world is not static but comes from a config file.
May 27, 2021
On Thursday, 27 May 2021 at 12:58:28 UTC, Christian Köstlin wrote:

> That looks nice, but unfortunately my data for servers and users in the real world is not static but comes from a config file.

Okay, but then parametrizing the static lambda with runtime parameters should work. The important fact is that the closure needs to be static.
May 27, 2021
On 2021-05-27 15:00, sighoya wrote:
> On Thursday, 27 May 2021 at 12:58:28 UTC, Christian Köstlin wrote:
> 
>> That looks nice, but unfortunately my data for servers and users in the real world is not static but comes from a config file.
> 
> Okay, but then parametrizing the static lambda with runtime parameters should work. The important fact is that the closure needs to be static.
Ah thanks, now I understand.
So what I came up with now is a combination of the things mentioned:

```D
import std;

string doSomething(string[] servers, string user) {
    return user ~ servers[0];
}
struct UserWithServers {
    string user;
    string[] servers;
}
void main(string[] args) {
    auto servers = args;
    auto users = ["u1", "u2", "u3"];
    auto usersWithServers = users.map!(user => UserWithServers(user, servers)).array;

    static fn = function(UserWithServers user) => user.servers.doSomething(user.user);
    writeln(taskPool.amap!(fn)(usersWithServers));
}
```

Making also the example a little bit more "realistic" by using dynamic data for servers.

I would like to use auto fn, but somehow saying that its a function is not enough for dmd. From my understanding a function would never need a context?!?

Thanks a lot!
Christian

P.S.: I still do not get how to post formatted snippets with thunderbird to the newsgroup/forum :/
May 27, 2021
On 5/27/21 10:13 AM, Christian Köstlin wrote:
> P.S.: I still do not get how to post formatted snippets with thunderbird to the newsgroup/forum :/

It's not possible currently.

I keep posting all my thunderbird posts *as if* the forum will highlight them in the hopes that some day it will retroactively work.

But I don't know... I can reformat the posts in my head, so it's not so bad.

-steve
May 27, 2021
On 5/27/21 2:58 AM, Christian Köstlin wrote:

>      writeln(taskPool.amap!(user => servers.doSomething(user))(users));

Luckily, parallel() is a free-standing function that does not require a "this context". Is the following a workaround for you?

  auto result = new string[users.length];
  users.enumerate.parallel.each!(en => result[en.index] = servers.doSomething(en.value));
  writeln(result);

Ali


« First   ‹ Prev
1 2