November 09, 2012
On Thursday, 8 November 2012 at 22:44:26 UTC, Jonathan M Davis wrote:
> I honestly wish that in didn't exist in the language. The fact that it it's an alias two different attributes is confusing, and
> people keep using it without realizing what they're getting into.
> If scope worked correctly, you'd only want it in specific
> circumstances, not in general. And since it doesn't work correctly
> aside from delegates, once it _does_ work correctly, it'll break
> code all over the place, because people keep using in, because they like how it corresponds with out or whatever.

I agree that it may likely be a cause for future issues. I wouldn't remove it though, rather relax it to an alias for const only (yes, because I like how it corresponds with out (input only vs. output only) and especially because it is very short - this diff of 3 characters really make a difference in function signatures :D). That'd fortunately still be possible without breaking existing code.

So please generalize my countless mentionings of 'in ref' to 'const ref'. ;)
November 09, 2012
On 9 November 2012 00:44, Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> On Thursday, November 08, 2012 21:49:58 Manu wrote:
> > That's cute, but it really feels like a hack.
> > All of a sudden the debugger doesn't work properly anymore, you need to
> > step-in twice to enter the function, and it's particularly inefficient in
> > debug builds (a point of great concern for my industry!).
> >
> > Please just with the compiler creating a temporary in the caller space. Restrict is to const ref, or better, in ref (scope seems particularly important here).
>
> I honestly wish that in didn't exist in the language. The fact that it
> it's an
> alias two different attributes is confusing, and people keep using it
> without
> realizing what they're getting into.


I understand it's an alias for 'const scope', but what's this about scope
not working correctly? What's wrong with it?
It seems like precisely what you want in this case... you don't want a
temporary escaping the function you pass it to.
What's the problem?

If scope worked correctly, you'd only
> want it in specific circumstances, not in general. And since it doesn't
> work
> correctly aside from delegates, once it _does_ work correctly, it'll break
> code all over the place, because people keep using in, because they like
> how
> it corresponds with out or whatever.
>

I like how it promises that the thing I pass can't escape the callee, not that it's the natural english compliment to 'out'.


November 09, 2012
On Friday, November 09, 2012 10:33:45 Manu wrote:
> On 9 November 2012 00:44, Jonathan M Davis <jmdavisProg@gmx.com> wrote:
> > On Thursday, November 08, 2012 21:49:58 Manu wrote:
> > > That's cute, but it really feels like a hack.
> > > All of a sudden the debugger doesn't work properly anymore, you need to
> > > step-in twice to enter the function, and it's particularly inefficient
> > > in
> > > debug builds (a point of great concern for my industry!).
> > > 
> > > Please just with the compiler creating a temporary in the caller space. Restrict is to const ref, or better, in ref (scope seems particularly important here).
> > 
> > I honestly wish that in didn't exist in the language. The fact that it
> > it's an
> > alias two different attributes is confusing, and people keep using it
> > without
> > realizing what they're getting into.
> 
> I understand it's an alias for 'const scope', but what's this about scope
> not working correctly? What's wrong with it?
> It seems like precisely what you want in this case... you don't want a
> temporary escaping the function you pass it to.
> What's the problem?

It only works with delegates. In all other cases, it's ignored.

> If scope worked correctly, you'd only
> 
> > want it in specific circumstances, not in general. And since it doesn't
> > work
> > correctly aside from delegates, once it _does_ work correctly, it'll break
> > code all over the place, because people keep using in, because they like
> > how
> > it corresponds with out or whatever.
> 
> I like how it promises that the thing I pass can't escape the callee, not that it's the natural english compliment to 'out'.

There are cases where it's useful, but there are a ton of cases where it would be really annoying if it worked properly. For instance, in a _lot_ of cases, it would cause problems for a function taking an array, because it couldn't return a slice of the array. Any struct holding any reference types would be in the same boat, as would any class or AA.

In most cases, it really doesn't matter whether references escape, because the stuff is on the GC heap, and it'll collect it when it's appropriate. It would be a much bigger deal if you had to worry about ownership.

For delegates, it's great because it allows you to avoid allocating a closure, so if you don't need the delegate to escape, it makes perfect sense to make it scope. For everything else, whether it makes sense depends on what you're doing, and sometimes it would be useful, but most of the time, it really wouldn't be.

But since in currently only works for delegates, it's absolutely pointless to use it instead of const for anything else, and if/when scope is ever fixed to work with reference types in general, then all kinds of code that uses in will break, because it'll be escaping references, and the programmer didn't notice it. So, you might as well just not use it at this point except for delegates.

Regardless, you don't need in to use scope. You can just use scope directly. So, having in buys you _nothing_, and it unfortunately encourages people to use it (which IMHO is bad due to the issues with scope), because they like how it's the opposite of out. So, if we didn't have in, we'd be able to do exactly what we can do now, but people wouldn't be using scope all over the place via in just because they liked the idea of in being the opposite of out.

But originally, in comes from D1 (where it had a meaning similar to const IIRC, but scope had nothing to do with it), so it's been around for a long time, and there's pretty much no way that it would be removed from the language. I just think that it was mistake to have it given its current semantics. It's also a bit pointless IMHO to have an attribute which is an alias for other attributes. It's a needless duplication of functionality and just increases confusion.

- Jonathan M Davis
November 09, 2012
On 9 November 2012 10:46, Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> On Friday, November 09, 2012 10:33:45 Manu wrote:
> > On 9 November 2012 00:44, Jonathan M Davis <jmdavisProg@gmx.com> wrote:
> > > On Thursday, November 08, 2012 21:49:58 Manu wrote:
> > > > That's cute, but it really feels like a hack.
> > > > All of a sudden the debugger doesn't work properly anymore, you need
> to
> > > > step-in twice to enter the function, and it's particularly
> inefficient
> > > > in
> > > > debug builds (a point of great concern for my industry!).
> > > >
> > > > Please just with the compiler creating a temporary in the caller
> space.
> > > > Restrict is to const ref, or better, in ref (scope seems particularly
> > > > important here).
> > >
> > > I honestly wish that in didn't exist in the language. The fact that it
> > > it's an
> > > alias two different attributes is confusing, and people keep using it
> > > without
> > > realizing what they're getting into.
> >
> > I understand it's an alias for 'const scope', but what's this about scope
> > not working correctly? What's wrong with it?
> > It seems like precisely what you want in this case... you don't want a
> > temporary escaping the function you pass it to.
> > What's the problem?
>
> It only works with delegates. In all other cases, it's ignored.
>
> > If scope worked correctly, you'd only
> >
> > > want it in specific circumstances, not in general. And since it doesn't
> > > work
> > > correctly aside from delegates, once it _does_ work correctly, it'll
> break
> > > code all over the place, because people keep using in, because they
> like
> > > how
> > > it corresponds with out or whatever.
> >
> > I like how it promises that the thing I pass can't escape the callee, not that it's the natural english compliment to 'out'.
>
> There are cases where it's useful, but there are a ton of cases where it
> would
> be really annoying if it worked properly. For instance, in a _lot_ of
> cases,
> it would cause problems for a function taking an array, because it couldn't
> return a slice of the array.


Does that actually make sense? Surely a function that receives a scope argument can return that argument, since it's only passing it back to the same function that already owns it... it knows it can trust that function, since it was received from that function.

Any struct holding any reference types would be
> in the same boat, as would any class or AA.
>

I don't follow the problem with reference args. surely they can be evaluated just fine? Just that nothing can escape the function...

In most cases, it really doesn't matter whether references escape, because
> the
> stuff is on the GC heap, and it'll collect it when it's appropriate. It
> would
> be a much bigger deal if you had to worry about ownership.
>

It's true, and stack locals are perhaps the most important thing that scope could possibly protect... which makes it particularly applicable here.

For delegates, it's great because it allows you to avoid allocating a
> closure,
> so if you don't need the delegate to escape, it makes perfect sense to
> make it
> scope. For everything else, whether it makes sense depends on what you're
> doing, and sometimes it would be useful, but most of the time, it really
> wouldn't be.
>

Of course, and whether you apply the scope attribute to a function argument
equally depends on what you're doing :)
I still can't see any inherent problems from what you're saying. It's still
just an access restriction to enhance safety.

But since in currently only works for delegates, it's absolutely pointless
> to
> use it instead of const for anything else, and if/when scope is ever fixed
> to
> work with reference types in general, then all kinds of code that uses in
> will
> break, because it'll be escaping references, and the programmer didn't
> notice
> it. So, you might as well just not use it at this point except for
> delegates.
>

Very good point. I had no idea it didn't work. It'd be nice to see this fixed.

Regardless, you don't need in to use scope. You can just use scope directly.
> So, having in buys you _nothing_, and it unfortunately encourages people to
> use it (which IMHO is bad due to the issues with scope), because they like
> how
> it's the opposite of out.


The issue here is that 'scope' is an incomplete feature. I think it's
theoretically sound.
Every conversation/thing I've read on the topic certainly gave me the
impression it worked...

So, if we didn't have in, we'd be able to do exactly
> what we can do now, but people wouldn't be using scope all over the place
> via
> in just because they liked the idea of in being the opposite of out.
>
> But originally, in comes from D1 (where it had a meaning similar to const IIRC, but scope had nothing to do with it), so it's been around for a long time, and there's pretty much no way that it would be removed from the language.


Oh really? That's not at all what I had imagined. I presumed 'in' was invented when it was realised that 'const scope' was a super-common pattern. I still don't see any less value in it. Just that it should work properly.

I just think that it was mistake to have it given its current
> semantics. It's also a bit pointless IMHO to have an attribute which is an
> alias for other attributes. It's a needless duplication of functionality
> and
> just increases confusion.
>

Side topic: what I find myself constantly wanting is alias for attributes!
Especially when dealing with GDC/LDC which have their own non-standard
attributes. I'm often wanting to create a meaning alias for the set of
attributes applicable to my code.
And you can't currently use GDC attributes without static if and physically
duplicating all the attributed code >_<


November 09, 2012
Let me summarize my (final, I guess) proposal. I think it makes sense to compare it to C++ in order to anticipate and hopefully invalidate (mainly Andrei's) objections.

     parameter type     |   lvalue    |    rvalue
                        | C++     D   | C++     D
------------------------|-------------|------------
T                       | copy   copy | copy   move
T& / ref T              | ref    ref  | n/a    n/a
out T (D only)          |        ref  |        n/a
T&& (C++ only)          | n/a         | move
auto ref T (D only) (*) |        ref  |        ref
------------------------|-------------|------------
const T                 | copy   copy | copy   move
const T& / const ref T  | ref    ref  | ref    ref (*)
const T&& (C++ only)    | n/a         | move

(*): proposed additions

For lvalues in both C++ and D, there are 2 options: either copy the argument (pass-by-value) or pass it by ref. There's no real difference between both languages except for D's additional 'out' keyword and, with the proposed 'auto ref' syntax, an (imo negligible) ambiguity between 'ref T' and 'auto ref T' in D.
Rvalues are a different topic though. There are 3 possibilites in general: copy, move and pass by ref. Copying rvalue arguments does not make sense - the argument won't be used by the caller after the invokation, so a copy is redundant and hurts performance. D corrects this design flaw of C++ (which had to introduce rvalue refs to add move semantics on top of the default copy semantics) and therefore only supports moving instead. C++ additionally supports pass-by-ref of rvalues to const refs, but not to mutable refs. I propose to allow pass-by-ref to both const (identical syntax as C++, it's perfectly safe and logical) and mutable refs (new syntax with 'auto ref' to emphasize that the parameter may be an rvalue reference, with related consequences such as potentially missing side effects).

Regarding the required overloading priorities for the proposed additions to work properly, I propose:
1) lvalues: prefer pass-by-ref
   so: ref/out T -> auto ref T (*) -> const ref T -> (const) T
   - const lvalues:   const ref T -> (const) T
   - mutable lvalues: ref/out T -> auto ref T (*) -> const ref T -> (const) T
2) rvalues: prefer pass-by-value (moving: argument allocated directly on
   callee's stack (parameter) vs. pointer/reference indirection implied by
   pass-by-ref)
   so: (const) T -> auto ref T (*) -> const ref T (*)

Finally, regarding templates, I'm in favor of dropping the current 'auto ref' semantics and propose to simply adopt the proposed semantics for consistency and simplicity and to avoid excessive code bloating. That shouldn't break existing code I hope (unless parameters have been denoted with 'const auto ref T', which would need to be changed to 'const ref T').

Now please go ahead and shoot. :)
November 09, 2012
On Fri, 09 Nov 2012 16:26:21 -0000, martin <kinke@libero.it> wrote:
> Let me summarize my (final, I guess) proposal. I think it makes sense to compare it to C++ in order to anticipate and hopefully invalidate (mainly Andrei's) objections.
>
>       parameter type     |   lvalue    |    rvalue
>                          | C++     D   | C++     D
> ------------------------|-------------|------------
> T                       | copy   copy | copy   move
> T& / ref T              | ref    ref  | n/a    n/a
> out T (D only)          |        ref  |        n/a
> T&& (C++ only)          | n/a         | move
> auto ref T (D only) (*) |        ref  |        ref
> ------------------------|-------------|------------
> const T                 | copy   copy | copy   move
> const T& / const ref T  | ref    ref  | ref    ref (*)
> const T&& (C++ only)    | n/a         | move
>
> (*): proposed additions
>
> For lvalues in both C++ and D, there are 2 options: either copy the argument (pass-by-value) or pass it by ref. There's no real difference between both languages except for D's additional 'out' keyword and, with the proposed 'auto ref' syntax, an (imo negligible) ambiguity between 'ref T' and 'auto ref T' in D.
> Rvalues are a different topic though. There are 3 possibilites in general: copy, move and pass by ref. Copying rvalue arguments does not make sense - the argument won't be used by the caller after the invokation, so a copy is redundant and hurts performance. D corrects this design flaw of C++ (which had to introduce rvalue refs to add move semantics on top of the default copy semantics) and therefore only supports moving instead. C++ additionally supports pass-by-ref of rvalues to const refs, but not to mutable refs. I propose to allow pass-by-ref to both const (identical syntax as C++, it's perfectly safe and logical) and mutable refs (new syntax with 'auto ref' to emphasize that the parameter may be an rvalue reference, with related consequences such as potentially missing side effects).
>
> Regarding the required overloading priorities for the proposed additions to work properly, I propose:
> 1) lvalues: prefer pass-by-ref
>     so: ref/out T -> auto ref T (*) -> const ref T -> (const) T
>     - const lvalues:   const ref T -> (const) T
>     - mutable lvalues: ref/out T -> auto ref T (*) -> const ref T -> (const) T
> 2) rvalues: prefer pass-by-value (moving: argument allocated directly on
>     callee's stack (parameter) vs. pointer/reference indirection implied by
>     pass-by-ref)
>     so: (const) T -> auto ref T (*) -> const ref T (*)
>
> Finally, regarding templates, I'm in favor of dropping the current 'auto ref' semantics and propose to simply adopt the proposed semantics for consistency and simplicity and to avoid excessive code bloating. That shouldn't break existing code I hope (unless parameters have been denoted with 'const auto ref T', which would need to be changed to 'const ref T').
>
> Now please go ahead and shoot. :)

Nice detailed proposal.  I would suggest starting a new thread with it, to catch anyone who dozed off in this thread :p

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/
November 09, 2012
On Friday, 9 November 2012 at 16:53:01 UTC, Regan Heath wrote:
> Nice detailed proposal.  I would suggest starting a new thread with it, to catch anyone who dozed off in this thread :p

Thanks, and you're probably right. Here's the link:

http://forum.dlang.org/thread/zteryxwxyngvyqvukqkm@forum.dlang.org

I've elaborated on it a little bit to avoid redundant posts.
November 09, 2012
On Friday, November 09, 2012 15:55:12 Manu wrote:
> Does that actually make sense? Surely a function that receives a scope argument can return that argument, since it's only passing it back to the same function that already owns it... it knows it can trust that function, since it was received from that function.

It can't. That would mean that the reference escaped. That would be particularly deadly for delegates. Think about what happens if the scoped delegate is put into a struct which is returned.

struct Result
{
 delegate........ del;
 ...........
}

Result foo(scope delegate....... bar)
{
 ..........
 return Result(bar);
}

auto baz()
{
 Result r;
 {
 int n = 5;
 r = foo((){writeln(n);});
 }
 r.del();
}

baz has no idea where the delegate in r came from. It has no idea that it wasn't allocated as a closure. So, it's not going to allocate one, which means that the delegate refers to a part of the stack which won't exist anymore when the delegate actually gets called. If scope wasn't used, that wouldn't have been a problem, because a closure would have been allocated as soon as the delegate had been passed to foo, but because scope was used, it knows that the delegate won't escape, so it doesn't allocate the closure (since it's not necessary). But that only works because scope prevents escaping - including by the return value. So, the above code _must_ be invalid.

>> Any struct holding any reference types would be
> > in the same boat, as would any class or AA.
> 
> I don't follow the problem with reference args. surely they can be evaluated just fine? Just that nothing can escape the function...

It's the fact that they can't escape the function which is the problem. If scope is working correctly, than any and all reference types which are passed to the function via a scope parameter (be it directly or within another object) cannot escape the function in any way shape or form. So, you can't assign any of them to any static or module-level variables (not generally a big deal) and you can't use any of them in the return value (often a big deal). Sometimes, that's exactly what you want, but in the general case, you don't want to prevent anything you pass into a function from being returned from it, so scope quickly becames overly restrictive.

- Jonathan M Davis
November 10, 2012
On 9 November 2012 21:39, Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> On Friday, November 09, 2012 15:55:12 Manu wrote:
> > Does that actually make sense? Surely a function that receives a scope argument can return that argument, since it's only passing it back to the same function that already owns it... it knows it can trust that
> function,
> > since it was received from that function.
>
> It can't. That would mean that the reference escaped. That would be particularly deadly for delegates. Think about what happens if the scoped delegate is put into a struct which is returned.
>
> struct Result
> {
>  delegate........ del;
>  ...........
> }
>
> Result foo(scope delegate....... bar)
> {
>  ..........
>  return Result(bar);
> }
>
> auto baz()
> {
>  Result r;
>  {
>  int n = 5;
>  r = foo((){writeln(n);});
>  }
>  r.del();
> }
>
> baz has no idea where the delegate in r came from. It has no idea that it
> wasn't allocated as a closure. So, it's not going to allocate one, which
> means
> that the delegate refers to a part of the stack which won't exist anymore
> when
> the delegate actually gets called. If scope wasn't used, that wouldn't have
> been a problem, because a closure would have been allocated as soon as the
> delegate had been passed to foo, but because scope was used, it knows that
> the
> delegate won't escape, so it doesn't allocate the closure (since it's not
> necessary). But that only works because scope prevents escaping -
> including by
> the return value. So, the above code _must_ be invalid.
>

Okay, makes sense.

>> Any struct holding any reference types would be
> > > in the same boat, as would any class or AA.
> >
> > I don't follow the problem with reference args. surely they can be evaluated just fine? Just that nothing can escape the function...
>
> It's the fact that they can't escape the function which is the problem. If
> scope is working correctly, than any and all reference types which are
> passed
> to the function via a scope parameter (be it directly or within another
> object) cannot escape the function in any way shape or form. So, you can't
> assign any of them to any static or module-level variables (not generally a
> big deal) and you can't use any of them in the return value (often a big
> deal). Sometimes, that's exactly what you want, but in the general case,
> you
> don't want to prevent anything you pass into a function from being returned
> from it, so scope quickly becames overly restrictive.
>

I'm still not buying this. Here's a common struct I will pass by ref (perhaps the most common struct in my industry):

struct Vector { float, x,y,z,w; }
struct Matrix { Vector xRow, yRow, zRow, wRow; }

Vector mul( scope const ref Matrix m, scope const Vector v)
{
  Vector v;
  // perform a matrix multiply against the vector...
  // this work uses every single field of the inputs given, but the result
it produces has to references to the sources.
  // everything is operated on and copied to the output struct, which is
returned.
  return result;
}

Why should this be a problem?
The majority of my work-horse structs apply to this pattern. This is what I
imagine 'scope' to be for...
The main advantage I expect is that I can have confidence that passing
rvalues (temporaries) is safe, and that external code won't take references
to memory that I may not own/control. Is that not the point?

Surely the problem that scope should be protecting against is a pointer to any part of the argument escaping. *Copies* of values contained in the argument/s are fine.


November 10, 2012
On Saturday, November 10, 2012 13:21:42 Manu wrote:
> I'm still not buying this. Here's a common struct I will pass by ref (perhaps the most common struct in my industry):
> 
> struct Vector { float, x,y,z,w; }
> struct Matrix { Vector xRow, yRow, zRow, wRow; }
> 
> Vector mul( scope const ref Matrix m, scope const Vector v)
> {
>   Vector v;
>   // perform a matrix multiply against the vector...
>   // this work uses every single field of the inputs given, but the result
> it produces has to references to the sources.
>   // everything is operated on and copied to the output struct, which is
> returned.
>   return result;
> }
> 
> Why should this be a problem?
> The majority of my work-horse structs apply to this pattern. This is what I
> imagine 'scope' to be for...
> The main advantage I expect is that I can have confidence that passing
> rvalues (temporaries) is safe, and that external code won't take references
> to memory that I may not own/control. Is that not the point?
> 
> Surely the problem that scope should be protecting against is a pointer to any part of the argument escaping. *Copies* of values contained in the argument/s are fine.

Hmmmm. scope on value types is pointless, because there are no references to escape, but if you pass by ref, then it does become possible for a pointer to the argument to escape, but I don't know that that's actually actually covered by scope. The description for scope in docs is that "ref­er­ences in the pa­ ra­me­ter can­not be es­caped (e.g. as­signed to a global vari­able)." And taking the address of a local variable (which is the only way that any sort of reference to the data could escape) is never @safe anyway. If you passed in a pointer, and scope were fully working, then you'd be protected against the pointer escaping, but passing by ref isn't really the same thing. I'd have thought that taking the address of a variable passed by ref would fall into pretty much the same camp as taking the address of any other local variable, which is completely unsafe to escape to the point that I'm not sure that there's any point in protecting against it. It's just completely stupid to do anyway and is definitely @system. Outside of taking the address of a ref parameter, taking the address of a local variable and escpaing it is _always_ going to result in garbage, and ref parameters aren't really references in the normal sense, so I don't know.

You bring up a good point, but I don't know if it's applicable. Certainly, without the ref there (like is the case with the Vector that you're passing in), scope would never do anything, because it doesn't even theoretically have anything to do. It's purely a value type that's not even being passed by ref.

In general though, putting scope on struct parameters would cause a lot of problems, because of arrays that they might hold and whatnot. Slices wouldn't be able to escape (and so copies of the struct wouldn't be able escape without deep copying, let alone the array itself). So, while scope may be very useful in some such cases (assuming that it worked), it's not necessarily something that you'd want as a matter of course. Part of it probably depends on your programming style though. If you have a lot of functions that take arguments and don't return anything that was in them ever, then scope is less of a big deal, but that's the sort of thing that happens a _lot_ in my experience, so scope would very quickly become extremely annoying.

And actually, to make matters worse, I'm not sure that scope on delegates is working correctly. I thought that it was, but this code compiles:

import std.stdio;

void delegate() global;

void foo(scope void delegate() del)
{
    global = del;
}

void main()
{
    {
        char[5] bar = "hello";
        foo((){writeln(bar);});
    }
    char[7] baz = "goodbye";

    global();
}

It also prints out "hello", and if a closure had not been allocated, I would have at least half-expected it to print out "goodb", because I'd have thought that baz would have been taking up the same memory that bar had been. So, it looks like scope may be completely and utterly broken at this point. I don't know.

- Jonathan M Davis