On 24 March 2018 at 04:57, John Colvin via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Friday, 23 March 2018 at 22:01:44 UTC, Manu wrote:
>>
>> Forked from the x^^y thread...
>>
>> On 23 March 2018 at 12:16, Walter Bright via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>>
>>> On 3/23/2018 11:09 AM, Manu wrote:
>>>>
>>>>
>>>> [...]
>>>
>>>
>>> Rvalue references are not trivial and can have major unintended consequences. They're a rather ugly feature in C++, with weirdities. I doubt D will ever have them.
>>
>>
>> Can you please explain these 'weirdities'?
>> What are said "major unintended consequences"?
>> Explain how the situation if implemented would be any different than
>> the workaround?
>>
>> This seems even simpler than the pow thing to me.
>> Rewrite:
>>     func(f());
>> as:
>>     { auto __t0 = f(); func(__t0); }
>>
>
> I understand what you want, but I'm struggling to understand why it's such a huge deal.

Because it makes this kind of D code that interacts with C++
objectively worse than C++, and there's no reason for it.
You can't say to someone who just frustrated-ly doubled their line
count by manually introducing a bunch of temporaries in a tiny
function that appears to do something so simple as 'call a function',
that "oh yeah, isn't it cool that you can't just call your functions
anymore! isn't D cool! we should switch to D right?"
It's embarrassing. I've been put in the position where I have to try
and 'explain' this feature quite some number of times... they usually
just give me 'the look'™; ya know, quietly wondering if I'm still
sane, and all I end up with is someone who's about 95% less convinced
that D is cool than they were 5 seconds beforehand.
What pisses me off is that's such a pointless thing to happen, because
this issue is so trivial!

In my experience, people are evaluating how D will materially impact
the exact same code they're already writing in C++. This is one of
those ways that they will be materially impacted, and it's almost
enough all on its own to cause people to dismiss the entire thing on
the spot.
Pretty much the best case at this phase is that the D code is exactly
the same as C++. If we can trim off a few parens here and there
(ufcs?), maybe remove some '::' operators (modules that don't suck),
that's a huge win.


> The reason you want to pass by reference is for performance, to avoid copying the data at the call boundary.
>
> So there are 2 cases: an lvalue needs to be passed, or an rvalue needs to be passed.
>
> 1. The address of the lvalue is passed.
>
> 2. The rvalue is copied to a local, then the address of that local is passed.
>
> So in the rvalue case, you're not getting the performance benefit of passing by reference, because you have to copy to a local anyway.
>
> What I would do in D currently to get the same performance and API:
>
> void foo(float[32] v) { foo(v); }
> void foo(ref float[32] v) { ... }
>
> or
>
> void foo()(auto ref float[32] v) { ... }

Can't be extern(C++), can't be virtual either (both is likely).
I said before; you're talking about Scott Meyers 'universal
references' as a language concept, and I'm just talking about calling
a function.


> but I dont' get how or why. It's exactly D's solution to the problem.

It doesn't solve the problem... it doesn't even address the problem. You're talking about a totally different thing >_<


> There's a little more work to be done when thinking about extern(C++) and/or virtual functions, but most code for most people isn't made of virtual extern(C++) functions that take large value types can't accept the cost of copying a few lvalues.

Correct, as I said before; people are getting bent out of shape over a
thing that will likely never affect them!
This will likely have very little impact on normal D code because D
const, auto ref, D move semantics, etc. It will have large impact on
interaction with C++ code, and the impression D is able to make on
that set of users. They're an important target market.

On 24 March 2018 at 06:49, Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On 24.03.2018 05:03, Manu wrote:
>>
>> I have no idea what this paragraph means... can you elaborate further what you're talking about?
>
> This works:
>
> struct S{
>     int x;
>     void inc(){
>         this.x++; // note: 'this' is passed by ref
>     }
> }
>
> void main(){
>     S().inc();
> }
>
> but this does not:
>
> struct S{
>     int x;
> }
> void inc(ref S self){
>     self.x++; // note: 'self' is passed by ref
> }
>
> void main(){
>     S().inc();
> }
>
> I.e. there is a special case where your rewrite is already applied. Note how "inc" cannot even be made const.
>
> What I'm saying is that I don't really buy Jonathan's argument. Basically, you should just pass the correct arguments to functions, as you always need to do. If you cannot use the result of some mutation that you need to use, you will probably notice.

Your example demonstrates the exact reason why rvalue->ref is const&
in C++, and illegal in D though.
You mutate a temporary that times out at the end of the statement...
your statement is never assigned to anything, and has no effect.
If your statement is assigned to something, then you already have an
lvalue to pass to such a function that receives mutable ref.

On 24 March 2018 at 07:56, kinke via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Saturday, 24 March 2018 at 13:49:13 UTC, Timon Gehr wrote:
>>
>> What I'm saying is that I don't really buy Jonathan's argument. Basically, you should just pass the correct arguments to functions, as you always need to do. If you cannot use the result of some mutation that you need to use, you will probably notice.
>
>
> I agree, but restricting it to const ref would be enough for almost all of my use cases. The MS C++ compiler just emits a warning when binding an rvalue to a mutable ref ('nonstandard extension used'), I'd find that absolutely viable for D too.
>
>> There are only three sensible ways to fix the problem:
>>
>> 1. Just allow rvalue arguments to bind to ref parameters. (My preferred solution, though it will make the overloading rules slightly more complicated.)
>
>
> I always thought the main concern was potential escaping refs to the rvalue, which would be solvable by allowing rvalues to be bound to `scope ref` params only. That'd be my preferred choice as well.

I touched on this.
It sounds reasonable at first glance. But the reasoning goes:
1. for safety reasons, we won't allow implicit stack temporaries to
pass to functions that may escape them.
2. support passing temporaries only to 'scope ref'
3. realise that the implicit temp created by an rvalue is identical to
the manually authored temp (or any other stack locals in the caller),
you find that the only reasonable option to satisfy the alleged safety
concern, is that all stack variables may never be passed to any
function that's not 'scope ref'.

Ie, you can't do this for safety reasons, because the exact same
reasoning would exclude all stack variables ever from being passed the
same way. The proposition is self-defeating.
I was attracted to this idea for a short while, until I realised it
was ridiculous. ;)


>> 3. Continue to require code bloat (auto ref) or manual boilerplate
>> (overloads). (I'm not very fond of this option, but it does not require a
>> language change.)
>
>
> While `auto ref` seems to have worked out surprisingly well for code written in D, it doesn't solve the problem when interfacing with (many) external C++ functions taking structs (represented in D by structs as well) by (mostly const) ref. You're forced to declare lvalues for all of these args, uglifying the code substantially.

It's also not great for libraries. auto ref functions are template
functions... what if I want to export that function? I can't.
There are practical API and ABI reasons that you don't want every
function to be a template function.
Library authors carefully control which functions are 'real' functions
and which are templates, for a wide variety of reasons. Binary libs
are a thing. DLL's are a thing.

Here is what I've used if I had to: https://p0nce.github.io/d-idioms/#Rvalue-references:-Understanding-auto-ref-and-then-not-using-it

On Saturday, 24 March 2018 at 11:57:25 UTC, John Colvin wrote:
> I understand what you want, but I'm struggling to understand why it's such a huge deal.
>
> The reason you want to pass by reference is for performance, to avoid copying the data at the call boundary.

It's pretty simple:

float foo() { ... }
ref float bar() { ... }

void someFunc(ref float);

someFunc(bar()); // ok

float temp = foo();
someFunc(temp); // Have to create a temporary anyways for the function to work
someFunc(foo()); // Compile error, need to use the hideous code above


So there really isn't any performance penalty cause if you want to call that function you are going to have to create a temporary variable anyways, but now you can't just call the function in one line. It requires multiple lines and a temporary variable name.

This becomes especially horrible for math libraries:

void someFunc(ref Vector3) { ... }

someFunc(a + b); // Can't do this

Vector3 temp = a + b;
someFunc(temp); // Have to do this

So there's no performance penalty for what he is requesting, but allows for a cleaner syntax to do the exact same thing.

> So there are 2 cases: an lvalue needs to be passed, or an rvalue needs to be passed.
>
> 1. The address of the lvalue is passed.
>
> 2. The rvalue is copied to a local, then the address of that local is passed.
>
> So in the rvalue case, you're not getting the performance benefit of passing by reference, because you have to copy to a local anyway.
>
> What I would do in D currently to get the same performance and API:
>
> void foo(float[32] v) { foo(v); }
> void foo(ref float[32] v) { ... }
>
> or
>
> void foo()(auto ref float[32] v) { ... }
>
> What is so totally unacceptable about those solutions? I personally like the second because it scales better to multiple parameters. I know you have said it's not relevant and annoying that people bring up auto ref, but I dont' get how or why. It's exactly D's solution to the problem.

It doesn't scale better, that's part of the problem:

void foo()(auto ref MyType1, auto ref MyType2, auto ref MyType3, auto ref MyType4) { ... }

The above has the possibility of generating 16 different functions that basically all do the exact same thing. It creates excessive bloat, and now what if you want to take the address of the function? You can't cause you have to choose between one of the 16 variants.

This is template bloat at it's finest, except it isn't even doing anything useful. I can only imagine telling someone they have to code gen 16 identical functions just to be able to call a function without having to create useless temporary variables in the scope a function is being called in.

On Saturday, 24 March 2018 at 17:30:35 UTC, Manu wrote:
> On 24 March 2018 at 04:57, John Colvin via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>> On Friday, 23 March 2018 at 22:01:44 UTC, Manu wrote:
>>>
>>> Forked from the x^^y thread...
>>>
>>> On 23 March 2018 at 12:16, Walter Bright via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>>>
>>>> On 3/23/2018 11:09 AM, Manu wrote:
>>>>>
>>>>>
>>>>> [...]
>>>>
>>>>
>>>> Rvalue references are not trivial and can have major unintended consequences. They're a rather ugly feature in C++, with weirdities. I doubt D will ever have them.
>>>
>>>
>>> Can you please explain these 'weirdities'?
>>> What are said "major unintended consequences"?
>>> Explain how the situation if implemented would be any different than
>>> the workaround?
>>>
>>> This seems even simpler than the pow thing to me.
>>> Rewrite:
>>>     func(f());
>>> as:
>>>     { auto __t0 = f(); func(__t0); }
>>>
>>
>> I understand what you want, but I'm struggling to understand why it's such a huge deal.
>
> Because it makes this kind of D code that interacts with C++
> objectively worse than C++, and there's no reason for it.
> You can't say to someone who just frustrated-ly doubled their line
> count by manually introducing a bunch of temporaries in a tiny
> function that appears to do something so simple as 'call a function',
> that "oh yeah, isn't it cool that you can't just call your functions
> anymore! isn't D cool! we should switch to D right?"
> It's embarrassing. I've been put in the position where I have to try
> and 'explain' this feature quite some number of times... they usually
> just give me 'the look'™; ya know, quietly wondering if I'm still
> sane, and all I end up with is someone who's about 95% less convinced
> that D is cool than they were 5 seconds beforehand.
> What pisses me off is that's such a pointless thing to happen, because
> this issue is so trivial!
>
> In my experience, people are evaluating how D will materially impact
> the exact same code they're already writing in C++. This is one of
> those ways that they will be materially impacted, and it's almost
> enough all on its own to cause people to dismiss the entire thing on
> the spot.
> Pretty much the best case at this phase is that the D code is exactly
> the same as C++. If we can trim off a few parens here and there
> (ufcs?), maybe remove some '::' operators (modules that don't suck),
> that's a huge win.
>
>
>> The reason you want to pass by reference is for performance, to avoid copying the data at the call boundary.
>>
>> So there are 2 cases: an lvalue needs to be passed, or an rvalue needs to be passed.
>>
>> 1. The address of the lvalue is passed.
>>
>> 2. The rvalue is copied to a local, then the address of that local is passed.
>>
>> So in the rvalue case, you're not getting the performance benefit of passing by reference, because you have to copy to a local anyway.
>>
>> What I would do in D currently to get the same performance and API:
>>
>> void foo(float[32] v) { foo(v); }
>> void foo(ref float[32] v) { ... }
>>
>> or
>>
>> void foo()(auto ref float[32] v) { ... }
>
> Can't be extern(C++), can't be virtual either (both is likely).
> I said before; you're talking about Scott Meyers 'universal
> references' as a language concept, and I'm just talking about calling
> a function.
>
>
>> but I dont' get how or why. It's exactly D's solution to the problem.
>
> It doesn't solve the problem... it doesn't even address the problem. You're talking about a totally different thing >_<
>

Auto ref allows the unnecessary copy to be avoided for lvalues and creates a temporary (as part of passing the value) for rvalues. It has downsides (virtual functions and extern(C++), but it does directly address the problem you're talking about, unless I have totally misunderstood you.

Here is a small proof of concept I made to demonstrate how easy it seems to be to use `auto ref` to call a C++ virtual const& function without incurring any more copies than would happen with the same calls from C++. I'm sure it could be improved a lot, but does the basic concept match what you would need?

// D source file:

/// mix this in to your extern(C++) class with a list of the functions where you
/// want to be able to pass rvalues to ref parameters.
auto rValueRefCalls(Funcs ...)()
{
    string ret;
    foreach (foo; Funcs)
        ret ~= `extern(D) void ` ~ __traits(identifier, foo) ~ `(Args ...)(auto ref Args args)
    if (__traits(compiles, (&this.` ~ __traits(identifier, foo) ~ `)(args)))
    {
        (&this.foo)(args);
    }`;

    return ret;
}

extern(C++)
{
    class A
    {
        void foo(const ref int v);
        mixin(rValueRefCalls!foo);
    }

    A makeA();
}

void main()
{
    int x = 3;
    auto a = makeA();
    a.foo(x);
    a.foo(3);
}


// C++ source file:

#include<cstdio>

class A
{
public:
    virtual void foo(const int& v)
    {
        printf("%d\n", v);
    }
};

A *makeA()
{
    return new A;
}

On Saturday, 24 March 2018 at 23:03:36 UTC, John Colvin wrote:
> Auto ref allows the unnecessary copy to be avoided for lvalues and creates a temporary (as part of passing the value) for rvalues. It has downsides (virtual functions and extern(C++), but it does directly address the problem you're talking about, unless I have totally misunderstood you.

You are trading syntax bloat for binary bloat. Having 4 parameters with auto ref can generate 16 different variants for the exact same function. If you are doing any sort of mathematical calculations you could cause a potential cache miss for calling the same exact function because one of your parameters didn't end up being the same type of value. You are causing all this bloat and potential slowdowns all to avoid having to do this:

float value0 = a + b;
float value1 = c + d;
float value2 = e + f;
someFunc(value0, value1, value2);

That's an inherent flaw in design. You obviously agree that there is a problem, but how can you justify "auto ref" being a "good" solution to the problem? It's a horrible one, it causes excessive bloat and potential slowdowns just to get a cleaner readable syntax. That shouldn't be a solution that is deemed acceptable.

On Saturday, 24 March 2018 at 23:03:36 UTC, John Colvin wrote:
> Here is a small proof of concept I made to demonstrate how easy it seems to be to use `auto ref` to call a C++ virtual const& function without incurring any more copies than would happen with the same calls from C++. I'm sure it could be improved a lot, but does the basic concept match what you would need?
>
> // D source file:
>
> /// mix this in to your extern(C++) class with a list of the functions where you
> /// want to be able to pass rvalues to ref parameters.
> auto rValueRefCalls(Funcs ...)()
> {
>     string ret;
>     foreach (foo; Funcs)
>         ret ~= `extern(D) void ` ~ __traits(identifier, foo) ~ `(Args ...)(auto ref Args args)
>     if (__traits(compiles, (&this.` ~ __traits(identifier, foo) ~ `)(args)))
>     {
>         (&this.foo)(args);
>     }`;
>
>     return ret;
> }
>
> extern(C++)
> {
>     class A
>     {
>         void foo(const ref int v);
>         mixin(rValueRefCalls!foo);
>     }
>
>     A makeA();
> }
>
> void main()
> {
>     int x = 3;
>     auto a = makeA();
>     a.foo(x);
>     a.foo(3);
> }
>
>
> // C++ source file:
>
> #include<cstdio>
>
> class A
> {
> public:
>     virtual void foo(const int& v)
>     {
>         printf("%d\n", v);
>     }
> };
>
> A *makeA()
> {
>     return new A;
> }

That isn't going to scale to the number of potential functions that are going to need this. It's going to cause slow compile speeds and create a bloated binary.

On Sunday, March 25, 2018 00:34:38 Rubn via Digitalmars-d wrote:
> On Saturday, 24 March 2018 at 23:03:36 UTC, John Colvin wrote:
> > Auto ref allows the unnecessary copy to be avoided for lvalues and creates a temporary (as part of passing the value) for rvalues. It has downsides (virtual functions and extern(C++), but it does directly address the problem you're talking about, unless I have totally misunderstood you.
>
> You are trading syntax bloat for binary bloat. Having 4 parameters with auto ref can generate 16 different variants for the exact same function. If you are doing any sort of mathematical calculations you could cause a potential cache miss for calling the same exact function because one of your parameters didn't end up being the same type of value. You are causing all this bloat and potential slowdowns all to avoid having to do this:
>
> float value0 = a + b;
> float value1 = c + d;
> float value2 = e + f;
> someFunc(value0, value1, value2);
>
> That's an inherent flaw in design. You obviously agree that there is a problem, but how can you justify "auto ref" being a "good" solution to the problem? It's a horrible one, it causes excessive bloat and potential slowdowns just to get a cleaner readable syntax. That shouldn't be a solution that is deemed acceptable.

How good or bad it is depends on what you're doing and how many auto ref parameters there are in your code. If you're using it occasionally, it's not a problem at all, whereas if you're using it all over the place, you do get a lot of template bloat.

Regardless, John's point was that auto ref solves the problem of being able to call a function with both lvalues and rvalues without copying lvalues, and he didn't understand why anyone was trying to argue that it doesn't do that. And I agree with him on that point. It does not help with virtual functions or extern(C++), and it creates a lot of template bloat if it's used heavily, so there are downsides to it, but it _does_ solve the basic problem of being able to call a function with both lvalues and rvalues without copying the lvalues. It just doesn't solve it in a way that everyone considers acceptable.

auto ref also helps with forwarding refness, so it's useful for more than just avoiding copying lvalues, but the entire reason that auto ref was originally added to the language was to solve this exact problem. And maybe we need a different solution for some use cases (like virtual functions or cases where the template bloat is deemed unacceptable), but auto ref is in the language to solve this problem. So, much as at may make sense to argue that it's not a good solution to this problem, it really doesn't make sense to argue that it has nothing to do with it.

- Jonathan M Davis

On 03/24/2018 03:03 AM, Jonathan M Davis wrote:
> On Saturday, March 24, 2018 01:37:10 Nick Sabalausky  via Digitalmars-d
> wrote:
>> Why require the callee's author to add boilerplate? Just do it for all
>> ref params that are given an rvalue.
> 
> Because if the point of the function accepting its argument by ref is
> because it's going to mutate the argument, then it makes no sense for it to
> accept rvalues, 

1. That *isn't* always the core point of a function which takes a non-const ref argument.

2. It's the caller who decides whether or not the ref-result is needed, not the callee.

3. The frequent recurring complaints about no rvalue references are a testament that this is too common a use-case for the current "manually insert temporaries" workaround to be satisfactory.

4. If the whole point of disallowing rvalue references is to prevent accidents due to callers not realizing a param is being passed by ref (as it sounds like you're suggesting), then it's nothing but a broken half-solution, because it fails to provide that safety (and thus fails its own charter) when that same caller, once again not realizing a param is ref, passes an *lvalue* without expecting it to change. *If* the problem we want to solve here really is making sure a caller knows when a param is ref, we've failed, and the only way to actually accomplish it is the C# approach: Require callers to mark their ref args with "ref" and raise an error when they don't.