Discuss
  • Forwarding `return()` in generators

Forwarding `return()` in generators

# Axel Rauschmayer (10 years ago)

AFAICT, return() throwing an exception (versus performing a return) is necessary if you want to forward it like in the following code:

function* take(n, iterable) {
    let iterator = iterable[Symbol.iterator]();
    try {
        while (n > 0) {
            let item = iterator.next();
            if (item.done) {
                return item.value;
            }
            yield item.value;
            n--;
        }
    } catch (e) {
        if (e instanceof ReturnException) {
            iterator.return(e.returnValue); // forward
        }
        ···
    }
}

Seems important w.r.t. composability of generators.

# Bergi (10 years ago)

Axel Rauschmayer wrote:

AFAICT, return() throwing an exception (versus performing a return) is necessary

It's not necessary, it just might be a bit more complicated:

 function* take(n, iterable) {
     let iterator = iterable[Symbol.iterator]();
     n = +n; // make sure it's a number, so that n>0 does never throw
     try {
         while (n > 0) {
             let item = iterator.next();
             if (item.done) {
                 return item.value;
             }
             yield item.value;
             n--;
         }
     } catch (e) {
         iterator.throw(e);
     } finally {
         iterator.return();
     }
 }

(I assume calling throw or return on a finished iterator is simply ignored, I'll have to check this back with the spec). The above code also has the additional nice property that it call .return() on the iterator when n values have been taken out of it.

Bergi

# Axel Rauschmayer (10 years ago)

Right, it doesn’t look like one needs to know the returned value when forwarding return().

But: you need to guard against other ways of reaching finally. Maybe like this:

function* take(n, iterable) {
    let iterator = iterable[Symbol.iterator]();
    n = +n; // make sure it's a number, so that n>0 does never throw
    let forwardReturn = true;
    try {
        while (n > 0) {
            let item = iterator.next();
            if (item.done) {
                forwardReturn = false;
                return item.value;
            }
            yield item.value;
            n--;
        }
        forwardReturn = false;
    } catch (e) {
        forwardReturn = false;
        iterator.throw(e);
    } finally {
        if (forwardReturn) {
            iterator.return();
        }
    }
}

The above code also has the additional nice property that it call .return() on the iterator when n values have been taken out of it.

That’s not what all the other constructs in ES6 do: they only call return() if iteration stops abruptly.

Also missing from this code: checking whether the iterator actually has the methods return() and throw() and responding accordingly.

# Ron Buckton (10 years ago)

Is your goal to wrap a generator, as it seems you are propagating the exception of the caller by calling iterator.throw(). However, you do not seem to be propagating the sent value, so the protocol here isn’t fully implmeneted.

If you just want to iterate values (and don’t really care about the return value of the iterable or propagating a thrown exception, you could write:

function* take(n, iterable) {
  n |= 0;
  if (n <= 0) {
    return;
  }
  // let for..of call return()
  for (let value of iterable) {
    yield value;
    if (n-- <= 0) {
      return;
    }
  }
}

If you want to support the full communication channel of a generator, you could write:

function* take(n, iterable) {
  let iterator = iterable[Symbol.iterator]();
  let step = () => iterator.next();
  n |= 0;
  // try..finally outside of loop
  try {
    let sent;
    while (n > 0) {
      let { value, done } = step();
      if (done) {
        return value;
      }
      n--;
      // try..catch local to the yield
      try {
        sent = yield value;
        step = () => iterator.next(sent);
      }
      catch (e) {
        if (typeof iterator.throw === "function") {
          step = () => iterator.throw(e);
        }
        else {
          throw e;
        }
      }
    }
  }
  finally {
    if (typeof iterator.return === "function") {
      iterator.return();
    }
  }
}
# Bergi (10 years ago)

Axel Rauschmayer schrieb:

Right, it doesn’t look like one needs to know the returned value when forwarding return().

Yeah, to support the iterator protocol as yield* does it one would need to forward that value. That might be another good use case for a meta property. However, I haven't really grasped what the value is used for, so I guess passing undefined is just fine.

But: you need to guard against other ways of reaching finally.

Why? I intended to always reach finally, and always close the iterator. Also, simplicity ftw :-)

That’s not what all the other constructs in ES6 do: they only call return() if iteration stops abruptly.

Only because all the other constructs in ES6 try to exhaust the iterator. And when it's finished anyway, one doesn't need to close it. There is in fact no problem with calling .return() too often, it just doesn't do anything to completed generators.

Btw, your take function is the perfect example where a non-exhausted iterator should be closed as return prematurely - like a break in a for-of loop would.

Also missing from this code: checking whether the iterator actually has the methods return() and throw() and responding accordingly.

Yupp, that might need to be added as well. However, the current behaviour (without the forwardReturn = false) is not that bad: If throw doesn't exist, it tries the return from the finally case before throwing a type error. To implement it correctly, I guess it should be

 try {
     while (--n > 0) {
         var {done, value} = iterator.next();
         if (done) return item.value;
         yield item.value;
     }
 } catch (e) {
     if (typeof iterator.throw == "function") {
         iterator.throw(e);
         done = true;
     } else
         throw e;
 } finally {
     if (!done)
         iterator.return();
 }

Bergi

# Axel Rauschmayer (10 years ago)

Good point, if it is about iteration, only return() needs to be propagated.

# Axel Rauschmayer (10 years ago)

But: you need to guard against other ways of reaching finally.

Why? I intended to always reach finally, and always close the iterator. Also, simplicity ftw :-)

That’s not what all the other constructs in ES6 do: they only call return() if iteration stops abruptly.

Only because all the other constructs in ES6 try to exhaust the iterator. And when it's finished anyway, one doesn't need to close it. There is in fact no problem with calling .return() too often, it just doesn't do anything to completed generators.

True. I hadn’t thought of that.

Btw, your take function is the perfect example where a non-exhausted iterator should be closed as return prematurely - like a break in a for-of loop would.

Right. Always closing is much simpler. Otherwise, you’d have to check whether everything was exhausted or not.

# Brendan Eich (10 years ago)

Axel Rauschmayer wrote:

Right. Always closing is much simpler. Otherwise, you’d have to check whether everything was exhausted or not.

This is by design, FWIW.

# Axel Rauschmayer (10 years ago)

Right. Always closing is much simpler. Otherwise, you’d have to check whether everything was exhausted or not.

This is by design, FWIW.

Which is OK, I’m more worried about generators behaving differently in reaction to return() than, e.g., array iterators. With the latter, you can continue iterating if you break from a for-of loop, with the former, you (generally) can’t. Either behavior is OK, but it should be consistent: Is return() for optional clean-up (array iterators) or does it really close an iterator (generators)?

The way that return() is handled in generators (via try-catch) means that the clean-up action is called in both cases: generator exhaustion and generator closure. If you don’t use a generator then a clean-up action in return() is only called if the iterator is closed, not if it is exhausted. Obviously that is a minor technical detail and easy to fix (call the clean-up action when you return { done: true }), but it’s still an inconsistency (which wouldn’t exist if return() was called in either case).

# Brendan Eich (10 years ago)

Axel Rauschmayer wrote:

The way that return() is handled in generators (via try-catch) means that the clean-up action is called in both cases: generator exhaustion and generator closure. If you don’t use a generator then a clean-up action in return() is only called if the iterator is closed, not if it is exhausted. Obviously that is a minor technical detail and easy to fix (call the clean-up action when you return { done: true }), but it’s still an inconsistency (which wouldn’t exist if return() was called in either case).

We want return (Python 2.5+ close) to be optional, though. So an iterator whether implemented by a generator function or otherwise sees no difference -- provided in the generator function implementation you do not yield in a try with a finally. Forcing return from a not-exhausted generator parked at yield other than in try-with-finally does not run any more code in the generator function's body.

If you, the implementor of an iterator, want to handle close (pre-exhaustion break from a for-of construct), implement return. If you as implementor choose a generator function to implement your iterator, you'll want that try-finally.

In this sense, return for generator function, while provided as a method of generator iterators, is encoded in the body of the generator function at the implementor's discretion (via try-yield-finally -- arguably one big try-finally with the guts that use yield in the try block).

HTH,

# Axel Rauschmayer (10 years ago)

We want return (Python 2.5+ close) to be optional, though. So an iterator whether implemented by a generator function or otherwise sees no difference -- provided in the generator function implementation you do not yield in a try with a finally. Forcing return from a not-exhausted generator parked at yield other than in try-with-finally does not run any more code in the generator function's body.

return() being optional is true for arrays:

function twoLoops(iterable) {
    let iterator = iterable[Symbol.iterator]();
    for (let x of iterator) {
        console.log(x);
        break;
    }
    for (let x of iterator) {
        console.log(x);
        break;
    }
}

twoLoops(['a', 'b', 'c']);
// Output:
// a
// b

But it is not true for generators:

function* elements() {
    yield 'a';
    yield 'b';
    yield 'c';
}

twoLoops(elements());
// Output:
// a

That is a difference between iterators that you have to be aware of and that needs to be documented per iterable. It’d be great if all iterables were indeed the same in this regard.

# Bergi (10 years ago)

Axel Rauschmayer wrote:

a difference between iterators that you have to be aware of and that needs to be documented per iterable.

Alternatively, just test if (typeof iterator.return == "function") :-) But yes, awareness is required. Maybe we should dub "closeable iterators" as a subtype of the iterator interface?

It’d be great if all iterables were indeed the same in this regard.

What do you suggest, that array iterators should not be continuable? So we'd have .return() methods on ArrayIterators, StringIterators, MapIterators, and SetIterators, which sets the respective [[Iterated*]] internal property to undefined?

Bergi

# Axel Rauschmayer (10 years ago)

It’d be great if all iterables were indeed the same in this regard.

What do you suggest, that array iterators should not be continuable? So we'd have .return() methods on ArrayIterators, StringIterators, MapIterators, and SetIterators, which sets the respective [[Iterated*]] internal property to undefined?

Maybe I overstated, maybe documenting whether an iterable produces continuable iterators is enough, but it is something to be aware of, something that has to be explained in conjunction with the iteration protocol. Similarly: whether an iterable restarts iteration every time you call [Symbol.iterator]() (generators don’t restart, arrays do).

# Allen Wirfs-Brock (10 years ago)

On Mar 26, 2015, at 10:50 PM, Axel Rauschmayer <axel at rauschma.de> wrote:

return() being optional is true for arrays:

...

But it is not true for generators:

but it is...

function* elements() {
    yield 'a';
    yield 'b';
    yield 'c';
}

elements.prototype.return = undefined;
# Brendan Eich (10 years ago)

Axel Rauschmayer wrote:

That is a difference between iterators that you have to be aware of and that needs to be documented per iterable. It’d be great if all iterables were indeed the same in this regard.

You're right, of course -- my point was too narrow, I was addressing only the "can implement return handling in either a generator function body or a custom iterator" side of the coin.

# Brendan Eich (10 years ago)

Brendan Eich wrote:

Axel Rauschmayer wrote:

It’d be great if all iterables were indeed the same in this regard.

I didn't reply to this last sentence. I don't agree that all iterables (we don't control the universe of such things, so perhaps you mean all standardized or built-in iterables) vend iterators with return methods. It's rare to both break from a for-of loop and want to iterate more after, so you're in hard-cases-make-bad-law land right there.

Iterators are best used by getting fresh ones and consuming them till "done" (whether exhausted or not). return is required only if an iterator hangs onto a scarce-enough resource. Many (most?) iterators do not.