Discuss
  • Proposal: for-of-withas a way to provide a value to the generator

Proposal: for-of-withas a way to provide a value to the generator

# Marius Gundersen (7 years ago)

Generators today can both "send" a value and "receive" a value from their consumer, so they can act as an interactive iterable. See for example fibonacci with reset

While generators are "producers" of values, the for-of statement acts as a "consumer" of these values. But unfortunately the for-of cannot send values back to the generator:

for(const value of fibonacci()){
  console.log(value);
  //no way to reset it :(
}

I propose a for-of-with statement, like so:

let feedback = false;
for(const value of fibonacci() with feedback){
  console.log(value)
  feedback = value == 8;
}

Note that the variable has to be defined before the for-of-with statement, but the initial value will not be seen inside the generator until function.sent is implemented.

The babel output of the for-of-with statement would be the following:

var reset = false;
var _iteratorNormalCompletion = true;
var _didIteratorError = false;
var _iteratorError = undefined;

try {
  for (var _iterator = fibonacci()[Symbol.iterator](), _step;
!(_iteratorNormalCompletion = (_step = _iterator.next(reset)).done);
_iteratorNormalCompletion = true) {
    var value = _step.value;

    console.log(value);
    reset = value == 8;
  }
} catch (err) {
  _didIteratorError = true;
  _iteratorError = err;
} finally {
  try {
    if (!_iteratorNormalCompletion && _iterator.return) {
      _iterator.return();
    }
  } finally {
    if (_didIteratorError) {
      throw _iteratorError;
    }
  }
}

Marius Gundersen

# Naveen Chawla (7 years ago)

Currently you can set one or more variables outside the scope of the generator function during iteration, which the generator function can read. Is this so bad?

# Michał Wadas (7 years ago)

Communication by mutating shared variables is not a solution.

# Marius Gundersen (7 years ago)

I've created a gist with a more realistic example and a way to do this today: gist.github.com/mariusGundersen/985189540541189ca80f60b59fa343ac

Notice that the with keyword is followed by an expression, it's not limited to a variable binding. This is similar to how a for(let i=0; i<10; i++) statement has the i<10 expression as a test.

# jong chan moe (7 years ago)

I think it would be natural to use it with the continue statement.

for (let value of fibonacci()) {
   console.log(value);
   continue with value == 8;
}
# Isiah Meadows (7 years ago)

To be honest, I'm not too sold on either of these:

  1. The with <expr> reads like an expression evaluated only once.

  2. The continue with <expr> does not provide a way to send a value on the

first next call.

One idea is to use a glorified reducer, like this (simplified):

function pipe(gen, start, func) {
    var iter = gen[Symbol.iterator]()
    var current = iter.next(start)
    while (!current.done) {
        start = func(current.value)
        current = iter.next(start)
    }
    return current.value
}

But even that has its limits (inability to recover from errors, for example), and it's a pretty inelegant solution to this IMHO.

In general, if you find yourself using iterators like that, for ... of is quite possibly the most way to use them. Think of them as sync send/receive channels, where you send the argument, block during processing, and receive through the return value. Here's how you should be using them, if you must:

var fibs = fibonacci()
var result = fibs.next(0)

while (!result.done) {
    console.log(value)
    result = fibs.next(8)
}
# Isiah Meadows (7 years ago)

Edit: remove bad email

# Marius Gundersen (7 years ago)

One of the reasons I think for-of should handle this is because of the error handling. Closing an iterator is not trivial, and so having to write that code yourself is error prone.

I originally considered continue x; (and break x;), but this is already used for labels, but it seems continue with x; (and break with x;) will work, as the following throws a syntax error today: with: while(true){continue with;}. Only problem as you said Isaiah is that you can't send an initial value, but at least until function.sent proposal is accepted that is not a problem, as there is no way for the generator to receive the initial next value, so it's not really of much value today. And since the Iterarot.return() method takes an optional value as well, it makes sense to have support for break with x;

# Isiah Meadows (7 years ago)

Inline:

On Tue, Aug 22, 2017 at 3:36 AM, Marius Gundersen <gundersen at gmail.com> wrote:

One of the reasons I think for-of should handle this is because of the error handling. Closing an iterator is not trivial, and so having to write that code yourself is error prone.

Very true. Here's how I feel it could be addressed: you could create a utility function to close an iterator after it throws, like this:

function use(gen, func) {
    const iter = gen[Symbol.iterator]()
    let caught, error, result
    try {
        return result = func(iter)
    } catch (e) {
        caught = true
        error = e
    } finally {
        try {
            iter.return()
        } finally {
            if (caught) throw error
            if (result == null || typeof result !== "object" && typeof
result !== "function") {
                throw new TypeError("result must be an object")
            }
        }
    }
}

Ideally, though, we should have some sort of resource management system like what I proposed a while back 1, but that's pretty non-trivial for starters. There's also the issue of I/O being almost always async in idiomatic JS APIs.

I originally considered continue x; (and break x;), but this is already used for labels, but it seems continue with x; (and break with x;) will work, as the following throws a syntax error today: with: while(true){continue with;}. Only problem as you said Isaiah is that you can't send an initial value, but at least until function.sent proposal is accepted that is not a problem, as there is no way for the generator to receive the initial next value, so it's not really of much value today.

Except it kind of is. The iterator protocol does include optional throw and return methods 2, and IteratorClose(iterator, completion) uses return internally 3.

And since the Iterarot.return() method takes an optional value as well, it makes sense to have support for break with x;

Isiah Meadows me at isiahmeadows.com

Looking for web consulting? Or a new website? Send me an email and we can get started. www.isiahmeadows.com

# Isiah Meadows (7 years ago)

Edit: the use function should be using result = iter.return(), not result = func(iter). My bad.

Isiah Meadows me at isiahmeadows.com

Looking for web consulting? Or a new website? Send me an email and we can get started. www.isiahmeadows.com

# Marius Gundersen (7 years ago)

The problem with closing an iterator is not just about closing it when it throws, it's also about closing it when the consumer stops reading early (for example when you use break in a for-of loop). This can be done, as described in the article I linked to or in the gist I linked to. Both the for-of-with and the continue-with/break-with solutions will work just as fine with iterators as they will with async-iterators, so they will both work with async resource handling (this whole thread came about because I needed this in my own code, for a for-await-of loop ).

I think the continue-with/break-with syntax might be a better solution to this problem. It's slightly trickier to implement for a transpiler, and it does not cover the initial next value (but as I said this isn't supported without the Function.sent proposal anyways). The advantage of break-with is that it works with the Iterator.return() method, something the for-of-with doesn't. For-of loops can already send a value using throw() (by throwing an exception), and using the proposed continue-with and break-with statements it will also be able to send values using next() and return() too.

# James Browning (7 years ago)

I've always wanted a way to do something similar so that operators over coroutines are virtually the same as over iterables, although I don't know we need it to be part of a for-of loop, simply having a set of utilities that help with dealing with coroutines would be nice.

In particular just a set of utilities that expose the internal operators in particular GetIterator, IteratorNext and IteratorClose would be particularly nice as there's subtleties that aren't entirely obvious from the outside especially once async iterators are part of the spec.

For example this might look sound:

function getIterator(iterable) {
    if (typeof iterable[Symbol.iterator] === 'function') {
        return iterable[Symbol.iterator]()
    } else {
        throw new TypeError(`object is not iterable`)
    }
}

But in the spec it actually stores a reference to the original method (presumably in case the method self-deletes in a getter or something weird like that) so it's actually:

function getIterator(iterable) {
    const method = iterable[Symbol.iterator]
    if (typeof method === 'function') {
        return Reflect.apply(method, iterable, [])
    } else {
        throw new TypeError(`object is not iterable`)
}

Which while a rare difference may lead to some strange bugs, this feature is shared amongst all of those methods, other subtleties include for-of loops not passing any arguments to iterable.next and that objects must be checked for as IteratorNext results.

Async iterators will be even worse for these sort've bugs as anyone implementing things that use async iterators directly will need to be aware of AsyncFromSyncIterator in addition to everything from sync iterators.

Because these can be done in userland (assuming we ignore that AsyncFromSyncIterator will have to a custom class as the real AsyncFromSyncIterator is mostly just a spec device), I think I'll try to implement them as a library and see how that works out with a few utility functions like using which will auto close iterators after completion.