On Sunday, 4 October 2020 at 08:57:01 UTC, Walter Bright wrote:
> On 10/2/2020 2:57 PM, Daniel N wrote:
>> Also see the formally accepted DIP1021!
>> https://github.com/dlang/DIPs/blob/148001a963f5d6e090bb6beef5caf9854372d0bc/DIPs/accepted/DIP1021.md
>
> That picks up the obvious cases, but the more subtle ones require @live's Data Flow Analysis to find.

Yes, that's true.

My point is that we are already working towards solving the very same issue in a different context. DIP1021 is the first step, @live is the final step. It's nothing inherently wrong with "-preview=in" but as it's currently designed it depends on the solution of @live.

It's also possible to solve in simpler ways, since "-preview=in" is just an optimization, one could simply disable it when it's not trivial to deduce that it's safe to pass by ref. With every new advance in the compiler it could be made more clever...

On Saturday, 3 October 2020 at 05:02:36 UTC, Andrei Alexandrescu wrote:
> On 10/2/20 6:11 PM, Walter Bright wrote:
>> On 10/2/2020 10:31 AM, Steven Schveighoffer wrote:
>>> And this might not be true on a different compiler.
>> 
>> This is looking like a serious problem.
>
> They say one should choose one's fights wisely, so I spent some time
> pondering. I could just let this thread scroll by and not think twice
> about it. By next week, I may as well forget.
>
> But then I realized this is /exactly/ the kind of crap that we'll all
> scratch our heads six months from now, "How did this ever pass review?

I take offense to that.  I'd prefer if you'd moderate your tone please.

> Who approved this? How in the world did a group of competent,
> well-intended people, looked at this and said - yep, good idea. Let's."
>
> ???
>

*You* approved it.

https://github.com/dlang/dmd/pull/11000#issuecomment-675605193


> This glib take is EXTREMELY concerning:
>
>> In the general case, no. You can have two distinct pointers with the
>> same value, and there's nothing the frontend can do to detect it.
>> 
>> This scenario has been brought up during the review. I doubt it will,
>> in practice, be an issue though. This is not a common pattern, nor
>> does it seems useful. It rather looks like a code smell.
>
> Wait a SECOND! Are we really in the market of developing and deploying language features that come unglued at the slightest and subtlest misuse? We most certainly shouldn't.
>
> I sincerely congratulated Mathias and the other participants for working on this. It's an important topic. Knowing that all those involved are very good at what they do, and without having looked closely, I was sure they got something really nice going that avoids this absolutely blatant semantic grenade. And now I see this is exactly it - we got a -preview of a grenade. How is this possible? How can we sleep at night now?
>
> Again: take a step back and reconsider, why did this pass muster?
>
> This is important, folks. It's really important as parameter passing goes to the core of what the virtual machine does. You can't say, meh, let's just fudge it here, and whatever is surprising it's on the user.
>
> Please, we really need to put back the toothpaste in the tube here. I could on everybody's clear head here to reconsider this.

Frankly, I think you are making a mountain out of a molehill here.  You are imagining a problem that doesn't exist; and if one does find an issue, the fault lies with the DMD compiler and not the D language specification.  Though evidently having clearer wording in the spec benefits all.

If you read nothing more of this reply, at least finish up until the end of this paragraph.  Please hold fire until GDC and LDC have implemented this feature, then we can discuss the pitfalls that we've encountered with it.  Basing decisions on behaviors observed with DMD is not the right approach, and if you are currently finding the situation to be a mess, it is a mess of DMD's own doing.

Plucking a fitting example from a Phobos unittest that demonstrates the kind of things DMD let's people get away with:
```
RefCounted!int* p;
{
    auto rc1 = RefCounted!int(5);
    p = &rc1;
    assert(rc1 == 5);
    assert(rc1._refCounted._store._count == 1);
    auto rc2 = rc1;
    assert(rc1._refCounted._store._count == 2);
}
assert(p._refCounted._store == null);
```
Why is this not a compile-time error?  DMD just isn't punishing users enough for the buggy code they've written.

- - -

I don't really have the heart to go through and unpick all points raised in this thread, but I think it's worth sharing the three key conclusions I took away from reviewing the pull request:

1. This is behind a -preview flag, and so should be treated as experimental.  Nothing breaks by having it there.  Nothing breaks if it were to be suddenly removed without any deprecation cycle.

2. Aliasing was raised multiple times throughout the review, I even gave this example at time to demonstrate my concerns:
```
void bar(in int a, out int b) { }
int a = 42;
bar(a, a);
```
But ultimately, worrying about this is missing the point, as the problem is already present in the compiler even without `-preview=in`, and Walter is working on fixing it.  In the meantime, these sorts of cases are relatively trivial to pick up and can be added as warnings in GDC and LDC until the front-end implements the semantic guarantees.

3. There is no danger of ref/non-ref mismatches in ABI, because `in` parameters that are inferred `ref` are going to be passed in memory anyway.

In the following:
```
alias A = void delegate(in long);
alias B = void delegate(const long);
```
Either `long` always gets passed in memory, or always gets passed in registers, in both cases, they are always going to be covariant.  The same remains true whatever type you replace `long` with.

The one place where `in` parameters start to get interesting is rather at the call site.  I think it is best illustrated with the following:
```
struct Foo { this(this); }
void fun1(const Foo f) { }
void fun2(in Foo f) { }
```
The compiler already ensures that non-trivially copy-able types never end up in a situation where a temporary is needed.  So `Foo` is always passed by ref, otherwise there'd be a double copy done at the call site.

So once again, both are covariant as far as the callee is concerned.  But in the case of `fun2`, the caller does something different.  The copy constructor is elided entirely, and that is the crux of the optimization that is at play here.  If you've assumed anything else, you're assumptions are sorely misplaced.

- - -

I've also skimmed past a passing concern that the ABI between compilers would be different.  Well, let me rest assure you that DMD, GDC and LDC have never been compatible in the first place, so there's no point worrying about that now.  Though DMD is really the one at fault for all incompatibilities by choosing to have a non-standard calling convention for extern(D) code...

On Sunday, 4 October 2020 at 09:15:36 UTC, Joseph Rushton Wakeling wrote:
>
> Platform-dependent, yes -- but the behavioural difference here can be 100% anticipated from the language rules (and hence, from the code), no?
>
> Isn't the issue with `-preview="in"` as currently implemented that _there is no way_ for the reader of the code to anticipate when ref will be used and when not?

As it is an optimization, I think that's best left to the interpretation of the compiler/compiler author, however optimizations should never break the semantic guarantee.

```
void fun1(const long[16] a, ref long[16] b, in long[16] c, out long[16] d);
fun1(var, var, var, var);

void fun2(in long[16] a, int b);
fun2(var, var[0]);
```

In the above, all parameters are passed in memory as per ABI requirements.  The difference lies in whether or not `var` is passed by reference directly or a via a temporary copy.

When it comes to `in` parameters, I'm of the opinion that a temporary should be passed if the type is trivially copy-able.  But as an optimization, if we look at the signature and determine that it's not possible for any parameters to be aliasing each other, then why not pass the `in` parameter as `ref`?

On Sunday, 4 October 2020 at 14:58:13 UTC, Iain Buclaw wrote:
>
> void fun2(in long[16] a, int b);
> fun2(var, var[0]);
> ```
>
> In the above, all parameters are passed in memory as per ABI requirements.

(Except the `int`/`var[0]` parameter that I added in last minute ;-)

On Sunday, 4 October 2020 at 14:58:13 UTC, Iain Buclaw wrote:
> As it is an optimization, I think that's best left to the interpretation of the compiler/compiler author, however optimizations should never break the semantic guarantee.

What is the intended semantic guarantee in simple words that makes sense to an end user?

From what you say I assume it is something along the lines of:

«Has const scope semantics. Values may be passed as references. Parameter passing will not trigger sideeffects on the actual or formal parameters.»

However, do you also have this constraint:

«The caller is responsible for ensuring that the actual parameter does not alias with other parameters or globals available to the called function.»

And if not, what do you have?


The language spec is way too convoluted. This is C++ level of convolutedness:

«The parameter is an input to the function. Input parameters behaves as if they have the const scope storage classes. Input parameters may be passed by reference by the compiler. Unlike ref parameters, in parameters can bind to both lvalues and rvalues (such as literals). Types that would trigger a side effect if passed by value (such as types with postblit, copy constructor, or destructor), and types which cannot be copied, e.g. if their copy constructor is marked as @disable, will always be passed by reference. Dynamic arrays, classes, associative arrays, function pointers, and delegates will always be passed by value, to allow for covariance. If the type of the parameter does not fall in one of those categories, whether or not it is passed by reference is implementation defined, and the backend is free to choose the method that will best fit the ABI of the platform.»

Stuff like this ought to be an implementors note in an appendix.

On Sunday, 4 October 2020 at 14:58:13 UTC, Iain Buclaw wrote:
> On Sunday, 4 October 2020 at 09:15:36 UTC, Joseph Rushton Wakeling wrote:
>>
>> Platform-dependent, yes -- but the behavioural difference here can be 100% anticipated from the language rules (and hence, from the code), no?
>>
>> Isn't the issue with `-preview="in"` as currently implemented that _there is no way_ for the reader of the code to anticipate when ref will be used and when not?
>
> As it is an optimization, I think that's best left to the interpretation of the compiler/compiler author, however optimizations should never break the semantic guarantee.

I agree.

Throughout this thread, I notice the confounding of two different concepts which are both referred to as "passing by reference".

Please separate _very clearly_
(A) the concept of passing by reference as per D language semantics, from
(B) the concept of passing by reference on machine instruction level.
It would be horrible if A becomes platform/compiler dependent for `in` parameters.
But B is an implementation detail that is at the discretion of the compiler implementer about which the D language user has no say (as Iain already mentioned, there is no D ABI spec and DMD, LDC, and GDC all make different ABI choices).

My worry from reading this thread is that `-preview=in` is making A platform/compiler dependent, to give users the feel that they get some control over B. I very much hope I am wrong.

-Johan

On Sunday, 4 October 2020 at 18:02:12 UTC, Johan wrote:
> My worry from reading this thread is that `-preview=in` is making A platform/compiler dependent, to give users the feel that they get some control over B. I very much hope I am wrong.

What is missing is the details of what the user must and must not do in order to ensure that all compliant compilers (all possible compilers that adhere to the spec) produce executables that generate the same output.

That shouldn't be guesswork, but explicit.

On Sunday, 4 October 2020 at 16:10:42 UTC, Ola Fosheim Grøstad wrote:
> On Sunday, 4 October 2020 at 14:58:13 UTC, Iain Buclaw wrote:
>> As it is an optimization, I think that's best left to the interpretation of the compiler/compiler author, however optimizations should never break the semantic guarantee.
>
> What is the intended semantic guarantee in simple words that makes sense to an end user?
>
> From what you say I assume it is something along the lines of:
>
> «Has const scope semantics. Values may be passed as references. Parameter passing will not trigger sideeffects on the actual or formal parameters.»
>
> However, do you also have this constraint:
>
> «The caller is responsible for ensuring that the actual parameter does not alias with other parameters or globals available to the called function.»
>
> And if not, what do you have?
>

The spec only explicitly states that non-POD types must *always be passed by reference*.  So the only semantic guarantee is that copy constructors are elided.

Derived and user-defined data types (excluding structs and static arrays) are explicitly *always passed by value*.  They might still be passed in memory by ABI, but in that case, it results in a copy-to-temp.

As for everything else, unless there is a language requirement to pass by reference then the conservative approach would be to say "why bother?".  I don't immediately see a reason to do copy elision for trivial types unless it is provable that there'd be no difference in observable program behaviour.

On Sunday, 4 October 2020 at 18:02:12 UTC, Johan wrote:
>
> My worry from reading this thread is that `-preview=in` is making A platform/compiler dependent, to give users the feel that they get some control over B. I very much hope I am wrong.
>

If any bugs get raised against LDC saying as much, then it is well within your right to close them as wontfix, then raise a bug against DMD for a wrong-code issue.

On Sunday, 4 October 2020 at 19:26:56 UTC, Iain Buclaw wrote:
> On Sunday, 4 October 2020 at 16:10:42 UTC, Ola Fosheim Grøstad wrote:
>> On Sunday, 4 October 2020 at 14:58:13 UTC, Iain Buclaw wrote:
>>> As it is an optimization, I think that's best left to the interpretation of the compiler/compiler author, however optimizations should never break the semantic guarantee.
>>
>> What is the intended semantic guarantee in simple words that makes sense to an end user?
>>
>> From what you say I assume it is something along the lines of:
>>
>> «Has const scope semantics. Values may be passed as references. Parameter passing will not trigger sideeffects on the actual or formal parameters.»
>>
>> However, do you also have this constraint:
>>
>> «The caller is responsible for ensuring that the actual parameter does not alias with other parameters or globals available to the called function.»
>>
>> And if not, what do you have?
>>
>
> The spec only explicitly states that non-POD types must *always be passed by reference*.  So the only semantic guarantee is that copy constructors are elided.
>

Of course, I'm forgetting the other side of `in` due to everyone only focusing on the `ref` part. :-)

The answer is yes on your mention of const scope semantics.  If I'd be pressed to bullet point it, I'd put down the following.

When an parameter is annotation with `in`:
- The parameter is not modifiable.
- All memory reachable from the parameter can not be clobbered (overwritten).
- The parameter does not escape.
- Copy constructors are elided by passing by-ref.
- Other forms of copy elision may occur if doing so does not change program behaviour.

Is that simple enough for an end-user?