Hey all,

I have been working on creating a multi-threaded application, so I have a shared configuration object which hold several command line parameters (which I fill using getopt).

The problem is that I get deprecation warnings when trying to set numerical values:

/usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation: read-modify-write operations are not allowed for shared variables. Use core.atomic.atomicOp!"+="(*receiver, 1) instead.

using getopt with a shared object and boolean values seems completely broken:

/usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation not allowed on bool *receiver += 1
/usr/include/dmd/phobos/std/getopt.d(751,46): Error: template instance `std.getopt.handleOption!(shared(bool)*)` error instantiating
/usr/include/dmd/phobos/std/getopt.d(435,15):        6 recursive instantiations from here: getoptImpl!(string, shared(string)*, string, shared(string)*, string, shared(VSync)*, string, shared(ulong)*, string, shared(bool)*, string, shared(Verbose)*)

Is getopt not supposed to be used with shared structs ?

On Friday, 11 May 2018 at 17:25:44 UTC, Danny Arends wrote:
> Hey all,
>
> I have been working on creating a multi-threaded application, so I have a shared configuration object which hold several command line parameters (which I fill using getopt).
>
> The problem is that I get deprecation warnings when trying to set numerical values:
>
> /usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation: read-modify-write operations are not allowed for shared variables. Use core.atomic.atomicOp!"+="(*receiver, 1) instead.
>
> using getopt with a shared object and boolean values seems completely broken:
>
> /usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation not allowed on bool *receiver += 1
> /usr/include/dmd/phobos/std/getopt.d(751,46): Error: template instance `std.getopt.handleOption!(shared(bool)*)` error instantiating
> /usr/include/dmd/phobos/std/getopt.d(435,15):        6 recursive instantiations from here: getoptImpl!(string, shared(string)*, string, shared(string)*, string, shared(VSync)*, string, shared(ulong)*, string, shared(bool)*, string, shared(Verbose)*)
>
> Is getopt not supposed to be used with shared structs ?

small example:

import std.getopt : getopt;

struct Settings {
  size_t myvalue = 0;
}

shared(Settings) config;

int main(string[] args) {
  getopt(args, "m|myvalue", &(config.myvalue));
  return(0);
}

Changing size_t to bool, results in the compilation error mentioned in the previous post

On Friday, May 11, 2018 17:25:44 Danny Arends via Digitalmars-d-learn wrote:
> Hey all,
>
> I have been working on creating a multi-threaded application, so I have a shared configuration object which hold several command line parameters (which I fill using getopt).
>
> The problem is that I get deprecation warnings when trying to set numerical values:
>
> /usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation:
> read-modify-write operations are not allowed for shared
> variables. Use core.atomic.atomicOp!"+="(*receiver, 1) instead.
>
> using getopt with a shared object and boolean values seems completely broken:
>
> /usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation
> not allowed on bool *receiver += 1
> /usr/include/dmd/phobos/std/getopt.d(751,46): Error: template
> instance `std.getopt.handleOption!(shared(bool)*)` error
> instantiating
> /usr/include/dmd/phobos/std/getopt.d(435,15):        6 recursive
> instantiations from here: getoptImpl!(string, shared(string)*,
> string, shared(string)*, string, shared(VSync)*, string,
> shared(ulong)*, string, shared(bool)*, string, shared(Verbose)*)
>
> Is getopt not supposed to be used with shared structs ?

getopt is designed to be single-threaded. The keyword shared is not used a single type in that module. If you want to use shared with anything in D, you have three options:

1. Use core.atomic to atomically do stuff like increment a integer.

2. Use a mutex (or synchronized block) to protect access to a shared object. You lock the mutex before accessing the object (and before _all_ accesses to that object). Within that section of code, you cast away shared and operate on the object as thread-local. Then at the end of that section, before releasing the lock, you make sure that no thread-local references to the shared object remain, and then free the lock. e.g. something like

synchronized(mutex)
{
    auto tls = cast(MyObject)mySharedObject.

    //... do stuff...

} // at this point, no thread-local references to mySharedObject
  // should // exist

In this scenario, the types in question are not designed to be used with shared at all. They're designed to be thread-local. So, if they're marked as shared, they're basically useless except when you protect them with a mutex and correctly cast away shared in order to operate on the object while it's protected by the mutex and therefore is thread-safe.

3. An object is designed to be used as shared. In this case, it can have shared member functions, and they can be called on shared objects, but then internally, the object has to deal with properly managing shared. It either uses atomics and/or uses mutexes as in #2 - it's just that in this case, it's done internally by the object rather than the programmer using the object.

synchronized classes are supposed to help with this particular case, but unfortunately, they're not fully implemented, so they don't currently help. But either way, the concept is still the same. You have on object that is designed to work with shared and deals with all of the appropriate protections internally.

So, those are your three options. In the case of getopt, if you want to use shared, you basically have to use #2. It's not dealing with a basic type like an int or pointer, so atomics aren't going to work, and it's not designed to work with shared. So, if you want to do anything with getopt and shared, you're going to have to protect it with a mutex.

That being said, I have to say that getopt seems like a really weird choice for wanting to do anything with shared. You normally call it immediately at the beginning of the program before doing anything with threads. The results might then be passed on to other threads via std.concurrency or through shared variables, but I wouldn't think that it would make a lot of sense to try and used getopt from more than one thread.

- Jonathan M Davis

On Friday, 11 May 2018 at 17:49:17 UTC, Jonathan M Davis wrote:
> On Friday, May 11, 2018 17:25:44 Danny Arends via Digitalmars-d-learn wrote:
>> [...]
>
> getopt is designed to be single-threaded. The keyword shared is not used a single type in that module. If you want to use shared with anything in D, you have three options:
>
> [...]

Hey Jonathan,

Thanks for the long and insightful answer.
The object is indeed constructed from the main thread, but afterwards multiple threads need to read the values given via the command line. since everything in the object is read only I was hoping to get away with making it shared.

I should just define tls variables to use with getopt and then set the corresponding variables in the shared object.

Danny

On Friday, May 11, 2018 18:01:18 Danny Arends via Digitalmars-d-learn wrote:
> On Friday, 11 May 2018 at 17:49:17 UTC, Jonathan M Davis wrote:
> > On Friday, May 11, 2018 17:25:44 Danny Arends via
> >
> > Digitalmars-d-learn wrote:
> >> [...]
> >
> > getopt is designed to be single-threaded. The keyword shared is not used a single type in that module. If you want to use shared with anything in D, you have three options:
> >
> > [...]
>
> Hey Jonathan,
>
> Thanks for the long and insightful answer.
> The object is indeed constructed from the main thread, but
> afterwards multiple threads need to read the values given via the
> command line. since everything in the object is read only I was
> hoping to get away with making it shared.
>
> I should just define tls variables to use with getopt and then set the corresponding variables in the shared object.

If you want to operate on that data as shared, then yes. But if you're really just looking for each thread to have its own copy, I'd suggest that you either give each thread its own copy on thread creation or pass it using std.concurrency rather than trying to deal with a shared variable - especially since the type system has no way to know that you're just planning to read from the shared variable after that, and it will scream at you for various operations - and once shared is fully locked down, it will probably scream if you do much of _anything_ with shared, since at that point, the compiler would only allow operations that were either marked with shared or where it could guarantee that they were thread-safe.

- Jonathan M Davis

On 5/11/18 1:25 PM, Danny Arends wrote:
> Hey all,
> 
> I have been working on creating a multi-threaded application, so I have a shared configuration object which hold several command line parameters (which I fill using getopt).
> 
> The problem is that I get deprecation warnings when trying to set numerical values:
> 
> /usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation: read-modify-write operations are not allowed for shared variables. Use core.atomic.atomicOp!"+="(*receiver, 1) instead.
> 
> using getopt with a shared object and boolean values seems completely broken:
> 
> /usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation not allowed on bool *receiver += 1
> /usr/include/dmd/phobos/std/getopt.d(751,46): Error: template instance `std.getopt.handleOption!(shared(bool)*)` error instantiating
> /usr/include/dmd/phobos/std/getopt.d(435,15):        6 recursive instantiations from here: getoptImpl!(string, shared(string)*, string, shared(string)*, string, shared(VSync)*, string, shared(ulong)*, string, shared(bool)*, string, shared(Verbose)*)
> 
> Is getopt not supposed to be used with shared structs ?
> 

No, just fill in a local copy, and then copy it to the shared location.

In the case where all you want is read-only access to the data, build it once and then cast to immutable:

immutable Config config;

void main(string[] args)
{
   Config localConfig;
   getopt(...); // fill in localConfig
   // VERY IMPORTANT, only do this ONCE, and don't use it before this line
   *(cast()&config) = localConfig;
}

It should now be accessible from all threads.

Normally, this is not encouraged (casting away immutable), but the compiler gets that an immutable global is allowed to be initialized once (in fact, you can do this without casting inside shared static constructors).

-Steve

On Friday, May 11, 2018 14:31:17 Steven Schveighoffer via Digitalmars-d- learn wrote:
> On 5/11/18 1:25 PM, Danny Arends wrote:
> > Hey all,
> >
> > I have been working on creating a multi-threaded application, so I have a shared configuration object which hold several command line parameters (which I fill using getopt).
> >
> > The problem is that I get deprecation warnings when trying to set numerical values:
> >
> > /usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation:
> > read-modify-write operations are not allowed for shared variables. Use
> > core.atomic.atomicOp!"+="(*receiver, 1) instead.
> >
> > using getopt with a shared object and boolean values seems completely broken:
> >
> > /usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation not
> > allowed on bool *receiver += 1
> > /usr/include/dmd/phobos/std/getopt.d(751,46): Error: template instance
> > `std.getopt.handleOption!(shared(bool)*)` error instantiating
> > /usr/include/dmd/phobos/std/getopt.d(435,15):        6 recursive
> > instantiations from here: getoptImpl!(string, shared(string)*, string,
> > shared(string)*, string, shared(VSync)*, string, shared(ulong)*, string,
> > shared(bool)*, string, shared(Verbose)*)
> >
> > Is getopt not supposed to be used with shared structs ?
>
> No, just fill in a local copy, and then copy it to the shared location.
>
> In the case where all you want is read-only access to the data, build it once and then cast to immutable:
>
> immutable Config config;
>
> void main(string[] args)
> {
>     Config localConfig;
>     getopt(...); // fill in localConfig
>     // VERY IMPORTANT, only do this ONCE, and don't use it before this
> line *(cast()&config) = localConfig;
> }
>
> It should now be accessible from all threads.
>
> Normally, this is not encouraged (casting away immutable), but the compiler gets that an immutable global is allowed to be initialized once (in fact, you can do this without casting inside shared static constructors).

Except that because this is done outside of a shared static constructor, you're technically mutating immutable data. This _should_ work, but it is violating the type system, and technically, the compiler is allowed to assume that config is Config.init everywhere. In practice, I wouldn't expect a problem, but because it's violating the type system, all bets are off.

- Jonathan M Davis

On 5/11/18 2:49 PM, Jonathan M Davis wrote:
> On Friday, May 11, 2018 14:31:17 Steven Schveighoffer via Digitalmars-d-
> learn wrote:
>> On 5/11/18 1:25 PM, Danny Arends wrote:
>>> Hey all,
>>>
>>> I have been working on creating a multi-threaded application, so I have
>>> a shared configuration object which hold several command line parameters
>>> (which I fill using getopt).
>>>
>>> The problem is that I get deprecation warnings when trying to set
>>> numerical values:
>>>
>>> /usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation:
>>> read-modify-write operations are not allowed for shared variables. Use
>>> core.atomic.atomicOp!"+="(*receiver, 1) instead.
>>>
>>> using getopt with a shared object and boolean values seems completely
>>> broken:
>>>
>>> /usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation not
>>> allowed on bool *receiver += 1
>>> /usr/include/dmd/phobos/std/getopt.d(751,46): Error: template instance
>>> `std.getopt.handleOption!(shared(bool)*)` error instantiating
>>> /usr/include/dmd/phobos/std/getopt.d(435,15):        6 recursive
>>> instantiations from here: getoptImpl!(string, shared(string)*, string,
>>> shared(string)*, string, shared(VSync)*, string, shared(ulong)*, string,
>>> shared(bool)*, string, shared(Verbose)*)
>>>
>>> Is getopt not supposed to be used with shared structs ?
>>
>> No, just fill in a local copy, and then copy it to the shared location.
>>
>> In the case where all you want is read-only access to the data, build it
>> once and then cast to immutable:
>>
>> immutable Config config;
>>
>> void main(string[] args)
>> {
>>      Config localConfig;
>>      getopt(...); // fill in localConfig
>>      // VERY IMPORTANT, only do this ONCE, and don't use it before this
>> line *(cast()&config) = localConfig;
>> }
>>
>> It should now be accessible from all threads.
>>
>> Normally, this is not encouraged (casting away immutable), but the
>> compiler gets that an immutable global is allowed to be initialized once
>> (in fact, you can do this without casting inside shared static
>> constructors).
> 
> Except that because this is done outside of a shared static constructor,
> you're technically mutating immutable data. This _should_ work, but it is
> violating the type system, and technically, the compiler is allowed to
> assume that config is Config.init everywhere. In practice, I wouldn't expect
> a problem, but because it's violating the type system, all bets are off.

It's not a problem. The compiler cannot assume the value is .init because it doesn't have access to the whole codebase to see if some static ctor has changed it. Effectively, you ARE changing it in a static ctor, but just after all the other static ctors have run.

Crucially noted as well: don't use this value inside a static ctor!

-Steve

On Friday, 11 May 2018 at 17:25:44 UTC, Danny Arends wrote:
> Hey all,
>
>
> Is getopt not supposed to be used with shared structs ?

I don't know but if you are opened to alternative i have just tested my getopt-like function at it works with shared, i don't know why... The design is different tho, it's not based on pointer, it rather generates the code to write the data without indirection.

links:
- https://github.com/BBasile/iz/commit/ac8b2e23214ec8c2d3a1000895b374f9c73f8ca7
- http://bbasile.github.io/iz/iz/options.html#handleArguments

Baz.