Discuss
  • Proposal: Add functionalities to the for/of protocol that take full advantange of the generators/iterators features

Proposal: Add functionalities to the for/of protocol that take full advantange of the generators/iterators features

# Claude Pache (12 years ago)

TL;DR: We propose to add functionalities to the for/of loop that make all features of generators (and iterators) available. Currently, the features unavailable in for/of loops are: values sent to the generator that are retrieved by the yield expression, and closing value produced by the generator using the return statement. The pretext is to make it possible to rewrite naturally someArray.reduce(functionExpression) et al. when used inside a generator function, so that functionExpression could contain yield statements, a problem raised in a recent thread.

In a recent thread [1], it has been noted that, when using .forEach and friends inside a generator, you cannot use yield inside the callback.

There has been some reflections on how to fix generators in order to avoid that defect; but it was taking the problem by the wrong side, for the issue does not come from the generator function, but from the .forEach construct: it uses a first-class callback function where for/of uses a second-class block.

Let's see how to reimplement .forEach et al. with ES6 tools, so that it just works. Consider a generic loop:

    // `arr` is an array
    for (let v of arr) {
        // do something with with v
    }

Suppose that you want to run the loop only for even-ranked items of the array. You could do something like this:

    Array.prototype.forEachEven = function(f, o) {
        // For simplicity, we ignore the issue of sparse arrays.
        for (let i = 0; i < this.length; i += 2) {
            f.call(o, this[i], i, this)
        }
    }

    arr.forEachEven(function(v) {
        // do something with v
        // But you can't use yield, break or return :-(
    }, this)

However, instead of transforming the entire for/of loop, it is more on-focus to act on the iteration part only. I mean the following:

    function* gEven(iterable) {
        let i = 0
        for (let x of iterable) {
            if (i % 2 === 0) {
                yield x
            }
            i += 1
        }
    }

    for (let v of gEven(arr)) {
        // do something with with v
        // You can use yield, break or return here, and it will just work :-)
    }

As you see, no more callback, no more problem. (For sure, you can also use yield* inside the loop in order to abstract away some code; but the focus of this message is on the iteration protocol.)

Now (and from here the things are becoming interesting), let us try to reimplement Array.prototype.some and Array.prototype.reduce using that pattern:

    r = arr.some(function(v) {
        let condition
        // compute `condition`
        return condition
    }, this)

    r = arr.reduce(function(x, v) {
        let combined
        // combine `x` and `v` into `combined`
        return combined
    }, initialValue)

Here, we need: (1) retrieve the intermediate results returned by the callback (condition and combined in our examples); (2) provide the final result of the loop (the value that'll be stored in r)

On the side of generators, all is already done, for you can: (1) retrieve an intermediate result by evaluating a yield expression; (2) produce the final result using a return statement. or, more generally, in the case of iterators: (1) an intermediate result is retrieved by the first argument of the next method; (2) the final result is produced by returning{ done: true, value: result }`.

Thus, we obtain:

    function* gSome(iterator) {
        for (let v of iterator) {
            if (yield v)
                return true
        }
        return false
    }

    function* gReduce(iterator, r) {
        for (let v of iterator) {
            r = yield [r, v]
        }
        return r
    }

However, on the side of the for/of loop, there is currently (to my knowledge) no way to send or retrieve results to/from the iterator being traversed. Therefore, I propose to add the functionality:

(1) Sending an intermediate result to the iterator: use a continue with expression statement. (2) Producing the final result: The for/of loop becomes an expression, which evaluates to either: (a) the closing value sent by the iterator, or: (b) the expression of a break with expression statement.

(There is no LineTerminator between continue/break and with. The use of the reserved word with means that it can't be confused with a label.) Using that syntax, our loops are written as following:

    r = for (let v of gSome(arr)) {
        let condition
        // compute `condition`, using `yield` ad libitum
        continue with condition
    }

    r = for (let [x, v] of gReduce(arr, initialValue)) {
        let combined
        // combine `x` and `v` into `combined`, using `yield` ad libitum
        continue with combined
    }

    r = for (let x of itr) {
        break with 42
    }
    // `r` will be equal to 42 if `itr` is not empty

Naturally, you can also use labels:

    break label with expression;
    continue label with expression;

It is true that the examples chosen are academic in the sense it isn't harder to write:

    var r = initialValue
    for (let v of arr) {
        let combined
        // combine `r` and `v` into `combined`, using `yield` ad libitum
        r = combined
    }

Netherveless, it is a shame that some nice features of iteration protocol are unavailable in for/of loops, which is probably intended to be the main consumer of iterators.

(And, if nothing else, it is a nice opportunity for the with keyword to redeem itself. :-P)

—Claude

[1] generators vs forEach: esdiscuss/2013-July/031919

# Andrew Fedoniouk (12 years ago)

Slightly different approach that I've found [1] quite useful.

Consider this simple function:

function range(start,end) { var i = start; return function() { if( i < end ) return i++; } }

It returns so called iterator function (not an Iterator but just a function).

and imagine that for/in ( or for/of )

for(var el in something)

statement is modified to support functions as collections. So body of the for loop is called until function something returns anything but not void.

Having this this we can write something like this:

for(var i in range(12,48)) { console.log(i); }

In principle this simple change eliminates need of most of generator/yield cases.

I am not sure if it close to the problem you describe but something tells me that it is.

[1] www.codeproject.com/Articles/33662/TIScript

# Rick Waldron (12 years ago)

On Tue, Jul 30, 2013 at 3:39 PM, Andrew Fedoniouk <news at terrainformatica.com

wrote:

Slightly different approach that I've found [1] quite useful.

Consider this simple function:

function range(start,end) { var i = start; return function() { if( i < end ) return i++; } }

It returns so called iterator function (not an Iterator but just a function).

and imagine that for/in ( or for/of )

for(var el in something)

statement is modified to support functions as collections. So body of the for loop is called until function something returns anything but not void.

Having this this we can write something like this:

for(var i in range(12,48)) { console.log(i); }

In principle this simple change eliminates need of most of generator/yield cases.

Which is exactly this:

function * range(start, end) { for (var i = start; i <= end; i++) { yield i; } }

for (var value of range(0, 9)) { console.log( value ); }

Or do you mean to say that generator functions and yield should be removed?

# Andrew Fedoniouk (12 years ago)

On Tue, Jul 30, 2013 at 1:43 PM, Rick Waldron <waldron.rick at gmail.com> wrote:

On Tue, Jul 30, 2013 at 3:39 PM, Andrew Fedoniouk <news at terrainformatica.com> wrote:

Slightly different approach that I've found [1] quite useful.

Consider this simple function:

function range(start,end) { var i = start; return function() { if( i < end ) return i++; } }

It returns so called iterator function (not an Iterator but just a function).

and imagine that for/in ( or for/of )

for(var el in something)

statement is modified to support functions as collections. So body of the for loop is called until function something returns anything but not void.

Having this this we can write something like this:

for(var i in range(12,48)) { console.log(i); }

In principle this simple change eliminates need of most of generator/yield cases.

Which is exactly this:

function * range(start, end) { for (var i = start; i <= end; i++) { yield i; } }

for (var value of range(0, 9)) { console.log( value ); }

Yes, it is close conceptually. With only exception: yield requires stack itself to be heap allocated thing. Many things around this.

In C++ I did very lightweight $generator/$yield implementation [1] but unfortunately that trick is not available in JS.

Or do you mean to say that generator functions and yield should be removed?

In principle, with functions-as-collections the yield and the whole generators stuff is not needed. If functions can be used on the right of 'in' or 'of' in 'for' then all 'yield' use cases that I saw so far can be implemented without the yield. So why do we need redundant entities?

[1] www.codeproject.com/Articles/29524/Generators

# Tab Atkins Jr. (12 years ago)

On Tue, Jul 30, 2013 at 2:10 PM, Andrew Fedoniouk <news at terrainformatica.com> wrote:

On Tue, Jul 30, 2013 at 1:43 PM, Rick Waldron <waldron.rick at gmail.com> wrote:

Or do you mean to say that generator functions and yield should be removed?

In principle, with functions-as-collections the yield and the whole generators stuff is not needed. If functions can be used on the right of 'in' or 'of' in 'for' then all 'yield' use cases that I saw so far can be implemented without the yield. So why do we need redundant entities?

You don't understand what yield does. It freezes the execution of the function at that point, waiting until .next() is called to resume execution. This means you can do a number of very convenient things, like yielding in the middle of a loop, or yielding within a try/catch block.

You can't do these with simple functions without significant refactoring. The convenience of yield is well-established by Python.

# Quildreen Motta (12 years ago)

In principle, with functions-as-collections the yield and the whole generators stuff is not needed. If functions can be used on the right of 'in' or 'of' in 'for' then all 'yield' use cases that I saw so far can be implemented without the yield. So why do we need redundant entities?

Semi-coroutines for asynchronous computations is one: killdream/promises-benchmark/blob/master/scenarios/serial/co.js#L49

# Andrew Fedoniouk (12 years ago)

On Tue, Jul 30, 2013 at 2:14 PM, Tab Atkins Jr. <jackalmage at gmail.com> wrote:

On Tue, Jul 30, 2013 at 2:10 PM, Andrew Fedoniouk <news at terrainformatica.com> wrote:

On Tue, Jul 30, 2013 at 1:43 PM, Rick Waldron <waldron.rick at gmail.com> wrote:

Or do you mean to say that generator functions and yield should be removed?

In principle, with functions-as-collections the yield and the whole generators stuff is not needed. If functions can be used on the right of 'in' or 'of' in 'for' then all 'yield' use cases that I saw so far can be implemented without the yield. So why do we need redundant entities?

You don't understand what yield does. It freezes the execution of the function at that point, waiting until .next() is called to resume execution. This means you can do a number of very convenient things, like yielding in the middle of a loop, or yielding within a try/catch block.

Trust me, I do understand what yield does. Check the link I've provided - it implements in C++ just what you said.

OK, here is an example that has multiple yielding exits:

function fruits() { var state = 0; return function() { switch(state++) { case 0: return "apple"; case 1: return "orange"; case 2: return "lime"; } } }

So if you will run this:

for(var fr in fruits) console.log( fr )

you will get apple orange lime

printed out.

Close enough to what you will do with yield.

Any yield use case can be reproduced this way.

You can't do these with simple functions without significant refactoring. The convenience of yield is well-established by Python.

Yep, but don't forget that Python uses reference counting. In Python generator creation and destruction is deterministic - you create it at the entry of for loop and it gets destroyed (or at least can be) at the end of it. In JS the only option is to make stack GC-able thing - generator gets created and live until the GC cycle.

Is this acceptable price for the feature - I really don't know. But I do know that for simple enumerators/iterators cases plain functions are more optimal in JS case. And they do not require such quite ugly syntax changes as "function*".

# Claude Pache (12 years ago)

Le 30 juil. 2013 à 21:39, Andrew Fedoniouk <news at terrainformatica.com> a écrit :

I am not sure if it close to the problem you describe but something tells me that it is.

You missed my point. I do not want to change the approach to iteration, even if it solves a problem. I am proposing additions around the for/of loop that exploits more completely the iteration protocol proposed for ES6, which has already matured lengthily (so I won't change it). The problem I've taken is merely an illustration for the concept.