http://d.puremagic.com/issues/show_bug.cgi?id=5660

           Summary: yield syntax sugar
           Product: D
           Version: D2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: DMD
        AssignedTo: nobody@puremagic.com
        ReportedBy: bearophile_hugs@eml.cc


--- Comment #0 from bearophile_hugs@eml.cc 2011-02-27 17:55:27 PST ---
Ranges are flexible and useful, but my practice with Python has shown me that many times you just want something simple that yields items lazily. To do this in D there is opApply() (that currently doesn't play well with most Phobos. Only std.array.array() and few other things are able to use them), but its syntax is awful, and even after years of usage I can't remember it and I need to look for an example of opApply() usage to copy & modify.

opApply() requires a significant amount of boilerplate code that makes code
quite longer, obfuscates the purpose of the code, and is bug-prone. This is a
simple example (D2 code, works with dmd 2.052):


// Program #1
import std.stdio;

/// Sequence of moves to solve Towers of Hanoi OEIS A001511
immutable final class hanoiTower {
    static opCall() {
        return new typeof(this)();
    }

    int opApply(int delegate(ref int) dg) {
        int result;
        int y = 1;

        result = dg(y);
        if (result)
            return result;

        foreach (x; hanoiTower()) {
            y = x + 1;
            result = dg(y);
            if (result)
                return result;

            y = 1;
            result = dg(y);
            if (result)
                return result;
        }

        return result;
    }

    int opApply(int delegate(ref int, ref int) dg) {
        int result;
        int i, y = 1;

        result = dg(i, y);
        i++;
        if (result)
            return result;

        foreach (x; hanoiTower()) {
            y = x + 1;
            result = dg(i, y);
            i++;
            if (result)
                return result;

            y = 1;
            result = dg(i, y);
            i++;
            if (result)
                return result;
        }

        return result;
    }
}


class genChar {
    char stop;

    this(char stop_) {
        stop = stop_;
    }

    static opCall(char stop_) {
        return new typeof(this)(stop_);
    }

    int opApply(int delegate(ref char) dg) {
        int result;
        for (char c = 'a'; c < stop; c++) {
            result = dg(c);
            if (result)
                return result;
        }
        return result;
    }

    int opApply(int delegate(ref int, ref char) dg) {
        int result, i;
        for (char c = 'a'; c < stop; c++) {
            result = dg(i, c);
            i++;
            if (result)
                return result;
        }
        return result;
    }
}

void main() {
    foreach (i, move; hanoiTower()) {
        writeln(i, " ", move);
        if (i >= 20)
            break;
    }
    writeln();

    foreach (i, c; genChar('h'))
        writeln(i, " ", c);
}


So I suggest a syntax like this, that's just syntax sugar for the precedent code:

// Program #2
import std.stdio;

yield(int) hanoiTower() {
    yield(1);
    foreach (x; hanoiTower()) {
        yield(x + 1);
        yield(1);
    }
}

yield(auto) genChar(char stop) {
    for (char c = 'a'; c < stop; c++)
        yield(c);
}

void main() {
    foreach (i, move; hanoiTower()) {
        writeln(i, " ", move);
        if (i >= 20)
            break;
    }
    writeln();

    foreach (i, c; genChar('h'))
        writeln(i, " ", c);
}


Different good modern used languages like Python, C# and Scala have a clean yield syntax, they show the way (if necessary I may translate this program #2 to those other three languages.)

The normal syntax of opApply() is of course kept in D, this is just an additive
change, that acts at syntax level (so just a lowering is needed to implement
it).

I suggest the generators with yield to be classes that define a "static opCall()" because their semantics is more flexible than structs. This causes a heap allocation, but situations where performance is so important are less common, and in such situation the programmer may just write the normal struct code with opApply(), or use a struct with more inlinable range protocol methods, or use something more lower level.

The syntax uses yield(...) as return type to denote such iterable classes, and
yield(auto) too supported, and it acts like auto return type for functions (all
yields in a generator must yield the same type).

Some disadvantages:
- "yield" may need to become a kind of keyword.
- Currently opApply() doesn't work well with std.algorithm and Phobos in
general. This is true for the normal current usage of opApply(), but this
proposal makes their usage more attractive and probably more widespread.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=5660


David Simcha <dsimcha@yahoo.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dsimcha@yahoo.com


--- Comment #1 from David Simcha <dsimcha@yahoo.com> 2011-02-27 18:08:32 PST ---
opApply plus fibers should do what you need.  It's inefficient, but so is pretty much any coroutine/yield-based way of doing things, including Python's. I was thinking a while back that something like this belonged in std.range, but I wanted to handle ref properly, so Bug 2443 got in my way.

Example (Warning:  Untested.):

/**
This must be a class or otherwise have reference/heap-allocated semantics.
*/
class OpApplyToRange(Iterable) {
    Fiber fiber;
    ForeachType!Iterable _front;
    bool _empty;
    Iterable iterable;

    void doLoop() {
        foreach(elem; iterable) {
            _front = elem;
            Fiber.yield();
        }

        _empty = true;
    }

    this(Iterable iterable) {
        this.iterable = iterable;
        fiber = new Fiber(&doLoop);
        fiber.call();
    }

    void popFront() {
        fiber.call();
    }

    ForeachType!Iterable front() @property { return _front; }

    bool empty() @property { return _empty; }
}

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=5660



--- Comment #2 from bearophile_hugs@eml.cc 2011-08-03 04:48:48 PDT ---
More programming in D shows me that syntax sugar like this one will be very useful to me:

yield(int) foo() {
    yield(1);
}


So, is it possible to automatically convert such kind of code into a Range (istead of opApply and fibers)?

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=5660



--- Comment #3 from bearophile_hugs@eml.cc 2011-08-03 05:08:13 PDT ---
Just as an example, this is Python2.6 code:


def process((i, j), a, b):
  if i == 0: yield (a, j)
  if j == 0: yield (i, b)
  if i == a: yield (0, j)
  if j == b: yield (i, 0)
  if j != b: yield (i + j - b, b) if (b < i + j) else (0, i + j)
  if i != a: yield (a, i + j - a) if (a < i + j) else (i + j, 0)



And this is how ShedSkin compiles it to C++ (a bit edited):


class __gen_process : public __iter<tuple2<int, int> *> {
public:
  tuple2<int, int> *__2, *__3;
  int a, b, i, j;

  int __last_yield;

  __gen_process(tuple2<int, int> *__2,int a,int b) {
    this->__2 = __2;
    this->a = a;
    this->b = b;
    __last_yield = -1;
  }

  tuple2<int, int> * __get_next() {
    switch(__last_yield) {
      case 0: goto __after_yield_0;
      case 1: goto __after_yield_1;
      case 2: goto __after_yield_2;
      case 3: goto __after_yield_3;
      case 4: goto __after_yield_4;
      case 5: goto __after_yield_5;
      default: break;
    }
    if ((i==0)) {
      __last_yield = 0;
      __result = (new tuple2<int, int>(2,a,j));
      return __result;
      __after_yield_0:;
    }
    if ((j==0)) {
      __last_yield = 1;
      __result = (new tuple2<int, int>(2,i,b));
      return __result;
      __after_yield_1:;
    }
    if ((i==a)) {
      __last_yield = 2;
      __result = (new tuple2<int, int>(2,0,j));
      return __result;
      __after_yield_2:;
    }
    if ((j==b)) {
      __last_yield = 3;
      __result = (new tuple2<int, int>(2,i,0));
      return __result;
      __after_yield_3:;
    }
    if ((j!=b)) {
      __last_yield = 4;
      __result = (((b<(i+j)))?((new tuple2<int, int>(2,((i+j)-b),b))):((new
tuple2<int, int>(2,0,(i+j)))));
      return __result;
      __after_yield_4:;
    }
    if ((i!=a)) {
      __last_yield = 5;
      __result = (((a<(i+j)))?((new tuple2<int, int>(2,a,((i+j)-a)))):((new
tuple2<int, int>(2,(i+j),0))));
      return __result;
      __after_yield_5:;
    }
    __stop_iteration = true;
  }
};

__iter<tuple2<int, int> *> *process(tuple2<int, int> *__2, int a, int b) {
  return new __gen_process(__2,a,b);
}

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=5660



--- Comment #4 from bearophile_hugs@eml.cc 2011-09-12 04:58:31 PDT ---
Alternative syntax (here inside the generator yield is used like the "return"
statement):


yield(int) hanoiTower() {
    yield(1);
    foreach (x; hanoiTower()) {
        yield x + 1;
        yield 1;
    }
}

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=5660



--- Comment #5 from bearophile_hugs@eml.cc 2011-09-12 04:59:15 PDT ---
Alternative syntax (here inside the generator yield is used like the "return"
statement):


yield(int) hanoiTower() {
    yield 1;
    foreach (x; hanoiTower()) {
        yield x + 1;
        yield 1;
    }
}

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=5660


Witold Baryluk <baryluk@smp.if.uj.edu.pl> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |baryluk@smp.if.uj.edu.pl


--- Comment #6 from Witold Baryluk <baryluk@smp.if.uj.edu.pl> 2012-02-14 21:18:51 PST ---
I have library for advanced iterators using Fibers. Take look here https://github.com/baryluk/iterators_2  (it is not update, but I use it very often, and locally have many updates to it, not just commited).

Library was created before D ranges was introduced, so few things should be improved, but it still works.

But any way - no need to syntax sugars.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=5660



--- Comment #7 from bearophile_hugs@eml.cc 2012-02-15 04:24:02 PST ---
(In reply to comment #6)

> But any way - no need to syntax sugars.

This was a request for syntax sugar for opApply. No need for Fibers here (fibers are useful, but here I am not asking for them).

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=5660


Adam D. Ruppe <destructionator@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |destructionator@gmail.com


--- Comment #8 from Adam D. Ruppe <destructionator@gmail.com> 2012-07-29 07:51:55 PDT ---
// this is your sugar!
string yield(string what) { return `if(auto result = dg(`~what~`)) return
result;`; }


Now, every time you would write

        result = dg(y);
        if (result)
            return result;

instead write:

mixin(yield("y"));

Or mixin(yield("i, y")); if you have multiple arguments.


And the boilerplate is way down. Since you keep result around,
you'll want to make it just if(result) instead of if(auto result),
but the idea is the same.


We might be able to make a class template or a mixn for
yield(int) hanoiTower() too, but writing int opApply(...)
isn't too bad.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=5660


Adrian Matoga <epi@atari8.info> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |epi@atari8.info


--- Comment #9 from Adrian Matoga <epi@atari8.info> 2012-10-12 05:46:18 PDT ---
Upvoted! Sometimes the yield syntax is the most natural and gives the most readable code. And it should be lowered to an input range, as this is often the only kind of iteration accepted by Phobos.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------