Consider this example simplified from this PR https://github.com/dlang/phobos/pull/6281

------
struct GetoptResult
{
    Option[] options;
}

struct Option
{
    string optShort;
    string help;
}

GetoptResult getopt(T...)(scope T opts) @safe
{
    GetoptResult res;
    auto o = Option(opts[0], opts[1]);
    res.options ~= o;
    return res;
}

void main() @safe
{
    bool arg;
    getopt("arg", "info", &arg);
}
------

$ dmd -dip1000 -run main.d

------
main.d(16): Error: scope variable o assigned to non-scope res
main.d(23): Error: template instance `onlineapp.getopt!(string, string, bool*)` error instantiating
------

The only way I've found to make the code compile and retain the pre-dip1000 behavior is to change the Option construction to

------
auto o = Option(opts[0].idup, opts[1].idup);
------

How can we return non-scoped result variables constructed from scope variables without copies?

On Wednesday, March 21, 2018 17:13:40 Jack Stouffer via Digitalmars-d wrote:
> Consider this example simplified from this PR https://github.com/dlang/phobos/pull/6281
>
> ------
> struct GetoptResult
> {
>      Option[] options;
> }
>
> struct Option
> {
>      string optShort;
>      string help;
> }
>
> GetoptResult getopt(T...)(scope T opts) @safe
> {
>      GetoptResult res;
>      auto o = Option(opts[0], opts[1]);
>      res.options ~= o;
>      return res;
> }
>
> void main() @safe
> {
>      bool arg;
>      getopt("arg", "info", &arg);
> }
> ------
>
> $ dmd -dip1000 -run main.d
>
> ------
> main.d(16): Error: scope variable o assigned to non-scope res
> main.d(23): Error: template instance `onlineapp.getopt!(string,
> string, bool*)` error instantiating
> ------
>
> The only way I've found to make the code compile and retain the pre-dip1000 behavior is to change the Option construction to
>
> ------
> auto o = Option(opts[0].idup, opts[1].idup);
> ------
>
> How can we return non-scoped result variables constructed from scope variables without copies?

The struct being returned would need to be marked with scope (or its members marked with scope) such that the compiler treated the result as containing values from the function arguments. I don't know whether that's possible with DIP 1000 as-is, and that could have some pretty nasty consquences when you consider how that then limits what can be done with the return value. Even if we _can_ mark GetoptResult as scope in some manner so that the copying isn't necessary, you're then just pushing the problem a level up. In this case, that's probably not a big deal, since this is stuff that's just going to be used in main and thrown away, but in the general case, having a struct that won't let you escape any of its members means that you're going to either need a struct with the same layout but without scope that you can copy the scope on to, or you're going to need to pull out each of the members individually to copy them or do whatever you need to do to work around scope.

My gut reaction is that issues along these lines will either prevent the use of scope when returning user-defined types as opposed to pointers or dynamic arrays, and/or they'll force the kind of copying that you're complaining about. But I don't think that I understand DIP 1000 well enough to know what it's limitations really are in cases like this. Hopefully, Walter has an answer.

However, I suspect that we're going to find cases where scope is a blunt enough instrument that we're going to be forced to either drop it or use @trusted in places (though in this case, using @trusted would be completely unreasonable, because you can't guarantee that it _isn't_ a problem for something from the caller to escape - not without examining the caller code, which would translate to marking the caller as @trusted, not the function being called). Actually, I'd be very surprised if we _didn't_ have cases like that. The question is how frequent those cases are and whether issues like this are going to pop up enough that it's going to usually make more sense to simply use pure and manually examine code to mark it as @trusted rather than use scope so that the compiler can mark a bunch of stuff as @safe for us. As long as the function doesn't return scope, we're probably fine, but once it starts returning scope, things start getting interesting.

In this particular case, it may make more sense to just let getopt be @safe on its own and just let the caller mark all of the uses of & as @trusted. I know that the bug report that sparked this is trying to make using getopt completely @safe without requiring the use of @trusted at all, but I don't think that being forced to allocate memory is worth that, and I don't think that mucking around with getopt to make it work with ref is worth that. There are advantages to getopt taking pointers, and I'd rather not see getopt's API change in a way that breaks code. If getopt itself is @safe, then it's trivial for the caller to determine that their code is @safe in spite of the use of & and thus either use @trusted appropriately or just not bother, since it's in main, which usually isn't doing much fancy. So, I _really_ don't think that making calling getopt inherently @safe is worth code breakage if it comes to that, and I don't think that it's worth allocating memory that we otherwise wouldn't have to allocate.

If it's possible to mark GetoptResult with scope such that we can use scope without copying, then great, but if it's not, then I'm inclined to argue that we should just make sure that getopt itself is @safe and not worry about whether the caller is doing anything @system to call getopt or not.

Regardless, this does raise a potential issue with scope and user-defined return types, and we should explore how possible it is for DIP 1000 to solve that problem without forcing copies that wouldn't be necessary in @system code.

- Jonathan M Davis

On Wednesday, 21 March 2018 at 17:13:40 UTC, Jack Stouffer wrote:
> Consider this example simplified from this PR https://github.com/dlang/phobos/pull/6281
>
> ------
> struct GetoptResult
> {
>     Option[] options;
> }
>
> struct Option
> {
>     string optShort;
>     string help;
> }
>
> GetoptResult getopt(T...)(scope T opts) @safe
> {
>     GetoptResult res;
>     auto o = Option(opts[0], opts[1]);
>     res.options ~= o;
>     return res;
> }
>
> void main() @safe
> {
>     bool arg;
>     getopt("arg", "info", &arg);
> }
> ------
>
> $ dmd -dip1000 -run main.d
>
> ------
> main.d(16): Error: scope variable o assigned to non-scope res
> main.d(23): Error: template instance `onlineapp.getopt!(string, string, bool*)` error instantiating
> ------
>
> The only way I've found to make the code compile and retain the pre-dip1000 behavior is to change the Option construction to
>
> ------
> auto o = Option(opts[0].idup, opts[1].idup);
> ------
>
> How can we return non-scoped result variables constructed from scope variables without copies?

I thought that maybe adding a function to Option and marking it as `scope` would work:

struct GetoptResult
{
    Option[] options;
    void addOptions(scope Option opt) @safe scope
    {
        //Error: scope variable opt assigned to non-scope this
        options ~= opt;
    }
}

But the compiler doesn't like that. However, I _did_ get it working by doing this:

GetoptResult getopt(T...)(scope T opts) @safe
{
    return GetoptResult([Option(opts[0], opts[1])]);
}

Which is not ideal, obviously, but the notion that some code has to be rewritten to accomodate ownership semantics is not a new one; one of the major complaints I've seen about Rust is that it requires you to adjust your coding style to satisfy the borrow checker.

On Wednesday, 21 March 2018 at 19:15:41 UTC, Meta wrote:
> But the compiler doesn't like that. However, I _did_ get it working by doing this:
>
> GetoptResult getopt(T...)(scope T opts) @safe
> {
>     return GetoptResult([Option(opts[0], opts[1])]);
> }
>
> Which is not ideal, obviously, but the notion that some code has to be rewritten to accomodate ownership semantics is not a new one; one of the major complaints I've seen about Rust is that it requires you to adjust your coding style to satisfy the borrow checker.

The problem here is that it's impossible to apply this to the actual getopt code :/

On Wednesday, 21 March 2018 at 18:50:59 UTC, Jonathan M Davis wrote:
> The struct being returned would need to be marked with scope (or its members marked with scope) such that the compiler treated the result as containing values from the function arguments. I don't know whether that's possible with DIP 1000 as-is

Not as far as I can tell. Marking it as scope in the function body means it can't be returned, and marking the array in GetoptResult as scope results in a syntax error.

> In this particular case, it may make more sense to just let getopt be @safe on its own and just let the caller mark all of the uses of & as @trusted.

This is thankfully the case currently.

> If it's possible to mark GetoptResult with scope such that we can use scope without copying, then great, but if it's not, then I'm inclined to argue that we should just make sure that getopt itself is @safe and not worry about whether the caller is doing anything @system to call getopt or not.
>
> Regardless, this does raise a potential issue with scope and user-defined return types, and we should explore how possible it is for DIP 1000 to solve that problem without forcing copies that wouldn't be necessary in @system code.

My cause for alarm with this limitation is this is one of the issues (taking address of locals safely) that DIP1000 was designed to solve. If, in practice, DIP1000 code can't be used for this case when the code become sufficiently complex, then we have a real problem on our hands.

On Wednesday, 21 March 2018 at 17:13:40 UTC, Jack Stouffer wrote:
> [snip]
>
> How can we return non-scoped result variables constructed from scope variables without copies?

If you re-wrote this so that it just had pointers, would it be simpler?

Below is my attempt, not sure it's the same...

struct Foo
{
    int b;
}

struct Bar
{
    Foo* a;
}

Bar bar(scope int* a) @safe
{
    Bar res;
    Foo x = Foo(*a);
    res.a = &x;
    return res;
}

void main() @safe
{
    int x = 1;
    bar(&x);
}

On Wednesday, March 21, 2018 19:50:52 Jack Stouffer via Digitalmars-d wrote:
> On Wednesday, 21 March 2018 at 18:50:59 UTC, Jonathan M Davis
>
> wrote:
> > The struct being returned would need to be marked with scope (or its members marked with scope) such that the compiler treated the result as containing values from the function arguments. I don't know whether that's possible with DIP 1000 as-is
>
> Not as far as I can tell. Marking it as scope in the function body means it can't be returned, and marking the array in GetoptResult as scope results in a syntax error.
>
> > In this particular case, it may make more sense to just let getopt be @safe on its own and just let the caller mark all of the uses of & as @trusted.
>
> This is thankfully the case currently.
>
> > If it's possible to mark GetoptResult with scope such that we can use scope without copying, then great, but if it's not, then I'm inclined to argue that we should just make sure that getopt itself is @safe and not worry about whether the caller is doing anything @system to call getopt or not.
> >
> > Regardless, this does raise a potential issue with scope and user-defined return types, and we should explore how possible it is for DIP 1000 to solve that problem without forcing copies that wouldn't be necessary in @system code.
>
> My cause for alarm with this limitation is this is one of the issues (taking address of locals safely) that DIP1000 was designed to solve. If, in practice, DIP1000 code can't be used for this case when the code become sufficiently complex, then we have a real problem on our hands.

Well, it certainly looks like once user-defined types are being returned, DIP 1000 falls short. And even if it were improved to allow for member variables to be marked with scope such that the compiler allowed for arguments to escape via a return value (which may or may not make sense with how DIP 1000 works), we then have the problem with potentially needing to copy the member variables (or the entire struct) in the caller - either that or to cast and have it be @trusted. I don't know how a big a problem this is really going to be in practice, but it certainly shows that DIP 1000 isn't a silver bullet.

- Jonathan M Davis