I'm working on an IRC bot as my pet project. I do know of Dirk[1]; I'm mostly doing this to learn the language.


For the sake of the argument, imagine three threads; one that reads from the socketstream of the socket connected to the IRC server, one Mediator that keeps track of what other threads and/or functions are interested in different IRC events (joins, parts, channel messages, ...), and one that simply wants to receive stuff to print to the screen.

The flow of events would go like this:
1. the 'reading' thread gets a raw string from the server
2. ...and passes it as a concurrency message to the Mediator
3. the Mediator receives it
4. ...and does serious regex voodoo to produce an IrcEvent struct, with various members detailing what type of event it was (again join, parts, ...), from and to whom, textual content if applicable, etc
5. the Mediator sends the struct as a concurrency message to our printing thread
6. ...which receives and prints it.
7. GOTO 1

The first concurrency message passed is a string, which works as expected. The second message is the translated struct of over 32 bytes in size (96 to be exact), and that's where things start breaking.

Passing by value, e.g. tid.send(eventStruct);
>core.exception.AssertError@/usr/include/d/4.8/std/variant.d(280): target must be non-null

Passing by immutable, e.g. tid.send(cast(immutable) eventStruct);
>/usr/include/d/4.8/std/variant.d:552: Error: cannot modify immutable expression *p
>/usr/include/d/4.8/std/concurrency.d:111: Error: template instance std.variant.VariantN!(32LU).VariantN.opAssign!(immutable(IrcEvent)) error instantiating

Passing an immutable pointer *initially* works, but then you get a race to finish using the struct before the scope it was declared in ends. (It's not on the heap.)
>received *(immutable(IrcEvent*)): immutable(IrcEvent)(UNSET, x"F0 30 E2 DA FF 7F 00 00 80 96 98 00 00 00 00 00 80 DF 9F 64 D2 7F 00 00 F0 30 E2 DA FF 7F 00 00 [...] Segmentation fault

From gdb;
>The value of '_param_1.sender' is a struct/class of type 'struct string' with the following fields:
>
>  length = 4409885 .. (Value of type 'ulong')
>     ptr = <Enter 1 to explore this field of type 'const char *'>
>
>Enter the field number of choice: 1
>'(_param_1.sender).ptr' is a pointer to a value of type 'const char'
>Continue exploring it as a pointer to a single value [y/n]: y
>'(_param_1.sender).ptr' a pointer pointing to an invalid memory location.


I put together http://dpaste.dzfl.pl/d7322971 earlier to demonstrate some of these errors, though I didn't mention the raciness of passing pointers there. To test that race I used http://dpaste.dzfl.pl/e6fd4569.


Are there any easy workarounds? Do people use std.concurrency, or is it largely avoided? Variant also seems to have some major issues... or maybe everything is caused by Variant and std.concurrency just exposes it.

I hit a wall here. :< Would love any tips or tricks you could spare!


[1]: https://github.com/JakobOvrum/Dirk

On Thursday, 8 August 2013 at 20:08:11 UTC, JR wrote:
> Passing by value, e.g. tid.send(eventStruct);
>>core.exception.AssertError@/usr/include/d/4.8/std/variant.d(280): target must be non-null

http://d.puremagic.com/issues/show_bug.cgi?id=9122, should be fixed now.

> Passing by immutable, e.g. tid.send(cast(immutable) eventStruct);
>>/usr/include/d/4.8/std/variant.d:552: Error: cannot modify immutable expression *p
>>/usr/include/d/4.8/std/concurrency.d:111: Error: template instance std.variant.VariantN!(32LU).VariantN.opAssign!(immutable(IrcEvent)) error instantiating

http://d.puremagic.com/issues/show_bug.cgi?id=10740

David

On 08/08/2013 01:08 PM, JR wrote:

> The first concurrency message passed is a string, which works as
> expected. The second message is the translated struct of over 32 bytes
> in size (96 to be exact), and that's where things start breaking.

I came across the following bug just the other day. It may be related:

"std.concurrency send() fails with structs over 32 bytes"

  http://d.puremagic.com/issues/show_bug.cgi?id=10740

> Passing an immutable pointer *initially* works, but then you get a race
> to finish using the struct before the scope it was declared in ends.

An immutable on the receiving side is a request to the caller that the data must be immutable. If you are casting data with mutable indirection to immutable, anything can happen.

immutable data is by nature not synchronized because it is read-only. (Of course you can synchronize if you know that it is casted data to begin with.)

Have you considered the 'shared' attribute? You are still responsible for dealing with race conditions.

> Are there any easy workarounds?

Messages are supposed to solve race conditions. Once a message gets some data nobody else should write to that data.

> Do people use std.concurrency, or is it largely avoided?

I have used it only in test code. I am not aware of major problems other than some Variant-related issues that you also mention.

Ali

On Thursday, 8 August 2013 at 20:08:11 UTC, JR wrote:
> I put together http://dpaste.dzfl.pl/d7322971 earlier to demonstrate some of these errors, though I didn't mention the raciness of passing pointers there. To test that race I used http://dpaste.dzfl.pl/e6fd4569.

That's just a bug in your code; when taking a pointer to stack data (which is un-@safe), you have to take care not to escape it from the scope.

> Are there any easy workarounds? Do people use std.concurrency, or is it largely avoided? Variant also seems to have some major issues... or maybe everything is caused by Variant and std.concurrency just exposes it.

Yep, the two issues are really just bugs in std.variant. std.concurrency itself is pretty nice and should be rather stable. The only major impediment is that 'shared' isn't really anywhere near useful right now, so you have to pretty much manually add/remove it when processing messages that contain mutable indirections (and hope you don't screw up in the process).

David

On Thursday, 8 August 2013 at 21:11:06 UTC, Ali Çehreli wrote:
> An immutable on the receiving side is a request to the caller that the data must be immutable. If you are casting data with mutable indirection to immutable, anything can happen.
>
> immutable data is by nature not synchronized because it is read-only. (Of course you can synchronize if you know that it is casted data to begin with.)
>
> Have you considered the 'shared' attribute? You are still responsible for dealing with race conditions.

Do you mean declaring the struct instance itself as shared and then passing its pointer? If so, I could not get receive() to catch it as anything other than a Variant, no matter how I tried.

> shared IrcEvent evt = p.parse(raw);
> tid.send(&evt);
> /* ... elsewhere ... */
> receive(
>     (shared IrcEvent* evt) {
>         writefln("%s: %s", typeof(evt).stringof, evt);
>     },
>     (Variant v) {
>         writefln("Variant: %s", v);  // <-- this is where it goes, as a shared(IrcEvent)*
>     }
> );

Sending it by (shared) value hits the same assertion in variant.d; target must be non-null. And if I still have to deal with the race if I go with a shared pointer, as you say, then it may be semantically more correct but my code will still be as broken. :>

I understand that the pointer goes invalid when the scope ends -- I just... don't know what other approach to take if I can't pass it by value.

I'm using gdc 4.8.1 (which is based on 2.062) and I would upgrade in a heartbeat, but this is the latest version available in Canonical's saucy repositories. It's not impossible to compile and package it myself, and if that's the only solution available then that's what I'll have to do. Iain Bucklaw seems to have a personal repository[1] but its packages are from 2010, so it seems I cannot ride on the success of others'. ;3


[1]: https://launchpad.net/~ibuclaw/+archive/ppa

On Thursday, 8 August 2013 at 21:16:36 UTC, David Nadlinger wrote:
> On Thursday, 8 August 2013 at 20:08:11 UTC, JR wrote:
>> I put together http://dpaste.dzfl.pl/d7322971 earlier to demonstrate some of these errors, though I didn't mention the raciness of passing pointers there. To test that race I used http://dpaste.dzfl.pl/e6fd4569.
>
> That's just a bug in your code; when taking a pointer to stack data (which is un-@safe), you have to take care not to escape it from the scope.

Can I manually store it on the heap and let the garbage collector keep it safe? I know of the opposite -- instantiating a class on the stack with the scope keyword -- but that doesn't help me here.

I guess I could have the sender wait for an 'okay done' priority message from the receiving thread before escaping the scope, but I don't have any guarantees that some other thread doesn't send a different message in the meantime (such as the server reader passing on a new raw string) when I'm not ready for such.

Also I could *technically* have a __gshared struct/class act as a container for an array of these event structs (with synchronized rotating push and front properties), but my pride would take a huge blow and I don't have enough ice cream.

On 08/09/2013 03:17 AM, JR wrote:

> On Thursday, 8 August 2013 at 21:16:36 UTC, David Nadlinger wrote:

>> when taking a pointer to stack data
>> (which is un-@safe), you have to take care not to escape it from the
>> scope.
>
> Can I manually store it on the heap and let the garbage collector keep
> it safe?

How about passing by-value? The following seems to work. (I used v2.064-devel-4203c06 but I don't know whether it is different on older compilers.)

// Receives by value:

	mixin MessageReaction.Print!(IrcEvent) immEvtPtr;

// Sends by value:

	IrcEvent e = fakeParser();
	p.send(e);

Ali

On Friday, 9 August 2013 at 15:45:51 UTC, Ali Çehreli wrote:
> How about passing by-value? The following seems to work. (I used v2.064-devel-4203c06 but I don't know whether it is different on older compilers.)
>
> // Receives by value:
>
> 	mixin MessageReaction.Print!(IrcEvent) immEvtPtr;
>
> // Sends by value:
>
> 	IrcEvent e = fakeParser();
> 	p.send(e);

I hit http://d.puremagic.com/issues/show_bug.cgi?id=9122 (http://dpaste.dzfl.pl/d7322971 line 107), but then my compiler is too old to include that bugfix.

>core.exception.AssertError@/usr/include/d/4.8/std/variant.d(280): target must be non-null

I'll look into compiling dmd.

On 08/10/2013 08:04 AM, JR wrote:

> I'll look into compiling dmd.

This project made it very simple for me:

  https://github.com/carlor/dlang-workspace

It builds dmd, druntime, and phobos by a single command:

  $ bash posix/gen.sh

It is a good idea to do a clean build after 'git pull's:

  $ bash posix/gen.sh clean
  $ bash posix/gen.sh

Ali

On Saturday, 10 August 2013 at 15:12:39 UTC, Ali Çehreli wrote:
> On 08/10/2013 08:04 AM, JR wrote:
> This project made it very simple for me:
>
>   https://github.com/carlor/dlang-workspace

My belated thanks; this worked great.