Why do generator expressions return generators?
On 9/6/2013 7:44 AM, Domenic Denicola wrote:
What does having these be generators buy us? I am almost sure I'm missing something.
I don't think you're missing anything. They seem to be more accurately described as Iterator Expressions than Generator Expressions. It might be interesting if you could use yield inside them, which would then make sending a value in useful. But without that they don't expose any generator interface externally.
Brandon Benvie wrote:
I don't think you're missing anything. They seem to be more accurately described as Iterator Expressions than Generator Expressions. It might be interesting if you could use yield inside them, which would then make sending a value in useful. But without that they don't expose any generator interface externally.
They could be called something else, for sure. The name works because they're sugar for a generator function immeidately invoked:
(for (x of [1,2, 3]) x * x)
is
(function*() { for (let x of [1,2, 3]) yield x * x; })()
The name also may have Python roots that predate Python 2.5's more complete (send as well as next; throw; close) generator interface.
I think we should keep the name, because it's more precise. Iterator expression could be taken to mean other things a bit too easily.
The name works because they're sugar for a generator function immeidately invoked:
I mean, that's true, but why is that true? What is the value of allowing you to send in values via .next(v), or send in exceptions via .throw(e)? Why not just make them sugar for creating custom iterable-iterators without shallow coroutine capabilities?
Le 06/09/2013 17:39, Brendan Eich a écrit :
They could be called something else, for sure. The name works because they're sugar for a generator function immeidately invoked:
There is also a non-generator desugaring (exercise left to the reader).
The name also may have Python roots that predate Python 2.5's more complete (send as well as next; throw; close) generator interface.
I think we should keep the name, because it's more precise. Iterator expression could be taken to mean other things a bit too easily.
I don't really care for the name, but I agree with Domenic that there is
no need for the expression to create a generator.
The presence of .next and .throw will be unnecessarily confusing to authors.
Domenic Denicola <mailto:domenic at domenicdenicola.com> September 6, 2013 7:44 AM
Why can't they just return iterable-iterators? ...
What does having these be generators buy us? I am almost sure I'm missing something.
The semantics is specified in terms of a generator function (non-escaping), with lexical environment inside it.
Anything else that's sane (i.e., that doesn't turn implicit let bindings in the for heads into heap properties of hidden objects) would involve desugaring into closures (nested if for-of heads nest) to keep the local let-bound variables alive.
The spec does not use desugaring, of course. While it could be done, in a debugger the results would differ observably from a generator function with let bindings. I think the spec should talk about this level of observability.
I keep forgetting how the ES6 draft uses "generator comprehension", not "generator expression" (the latter is a generator function expression). Contrast with Python.
Domenic Denicola <mailto:domenic at domenicdenicola.com> September 6, 2013 8:48 AM
I mean, that's true, but why is that true?
See my reply just sent.
What is the value of allowing you to send in values via
.next(v), or send in exceptions via.throw(e)?
An iterator does not reject .next calls that pass a value.
As for .throw, don't do that unless you want what you get. "Doctor, it hurts when I ...."
Why not just make them sugar for creating custom iterable-iterators without shallow coroutine capabilities?
Because we need the shallow continuation to hold the implicitly declared let bindings induced by the for heads.
From: Brendan Eich [brendan at mozilla.com]
The spec does not use desugaring, of course. While it could be done, in a debugger the results would differ observably from a generator function with let bindings. I think the spec should talk about this level of observability.
Hmm, getting somewhere here. Could you expand on this? E.g. give a code example where the observable results would be different if it returned a iteratable-iterator instead of a generator?
David Bruant <mailto:bruant.d at gmail.com> September 6, 2013 8:59 AM
There is also a non-generator desugaring (exercise left to the reader).
There's always a lambda encoding. So what? First, ES6 doesn't use desugaring to specify its semantics. Second, the closure or closures (plural, for nested for heads) cases are distinct in real implementations from a single generator with blocks. See message sent two back by me.
I don't really care for the name, but I agree with Domenic that there is no need for the expression to create a generator. The presence of
.nextand.throwwill be unnecessarily confusing to authors.
Now you are disagreeing with Domenic on .next. How pray tell would you have an iterator without .next?
If you mean that a useless argument can be passed into .next, that is allowed with any iterator too.
As for .throw being confusing, I doubt it. Throwing into something that can't catch is like using throw (the keyword) directly. Users can write all sorts of direct and indirect throws already.
Domenic Denicola <mailto:domenic at domenicdenicola.com> September 6, 2013 9:06 AM
Hmm, getting somewhere here. Could you expand on this? E.g. give a code example where the observable results would be different if it returned a iteratable-iterator instead of a generator?
In a debugger, I would see extra frames (without the debugger working to merge them to preserve the appearance of a generator with let blocks). Think about nested for-of heads, lifetimes of bindings. You need nested closures in general.
From: Brendan Eich [brendan at mozilla.com]
In a debugger, I would see extra frames (without the debugger working to merge them to preserve the appearance of a generator with let blocks). Think about nested for-of heads, lifetimes of bindings. You need nested closures in general.
Really? The Firefox and Chrome debuggers, from what I can see, always hide implementation stack frames from us. Are you referring to the C++ debugger you'd use while developing those engines?
Le 06/09/2013 18:06, Brendan Eich a écrit :
An iterator does not reject .next calls that pass a value.
fair enough (sorry about my earlier message on .next. It's part of the iterator interface, not sure why I wrote that)
As for .throw, don't do that unless you want what you get. "Doctor, it hurts when I ...."
Why exposing something that's meant to be a footgun? Just don't expose it?
Domenic Denicola <mailto:domenic at domenicdenicola.com> September 6, 2013 9:13 AM
Really? The Firefox and Chrome debuggers, from what I can see, always hide implementation stack frames from us.
This is not about "implementation stack frames". The frame or frames would be for your code written in the comprehension expression.
Are you referring to the C++ debugger you'd use while developing those engines?
We are probably in the weeds, but only slightly. The main thing is the spec does not "desugar", but if it did, desugaring to (potentially nested) closures would need to be specified instead of what's drafted now: generator function with let blocks.
Debuggers can't hide everything, and should not. It's awesome to black-box at every level of abstraction, Firefox's devtools support this -- it wins for users, never mind built-ins. But let's get back to the spec issue.
On 6 September 2013 18:04, Brendan Eich <brendan at mozilla.com> wrote:
The spec does not use desugaring, of course. While it could be done, in a debugger the results would differ observably from a generator function with let bindings. I think the spec should talk about this level of observability.
But it doesn't. No matter which way you spec it, AFAICT, the difference is not observable from within the (spec'ed) language. We don't spec a debugger. Moreover, such details will already be dependent on implementation and optimisation anyway. So that does not strike me as a compelling argument. (If the spec really cared to guide debuggers, some notes could take care of that, but I don't think it makes sense to even try.)
The price for not applying desugaring techniques is that we are pulling in a lot of unnecessary complexity and accidental inconsistency. I think it's quite a substantial cost, actually. Desugaring also is a great way of dog-fooding your own abstraction capabilities -- it can reveal language deficiencies quite relentlessly.
[I guess your "of course" activated my ranting trigger here, given that small core languages plus well-defined desugarings were still regarded best practice last time I looked. ;)]
Andreas Rossberg <mailto:rossberg at google.com> September 6, 2013 9:50 AM
But it doesn't. No matter which way you spec it, AFAICT, the difference is not observable from within the (spec'ed) language. We don't spec a debugger. Moreover, such details will already be dependent on implementation and optimisation anyway. So that does not strike me as a compelling argument. (If the spec really cared to guide debuggers, some notes could take care of that, but I don't think it makes sense to even try.)
Ok, noted. I would not be surprised if we do "go there", though.
The price for not applying desugaring techniques is that we are pulling in a lot of unnecessary complexity and accidental inconsistency. I think it's quite a substantial cost, actually. Desugaring also is a great way of dog-fooding your own abstraction capabilities -- it can reveal language deficiencies quite relentlessly.
I actually agree, and the simplest desugaring by far, much better than closures, for
(for (x of range(n)) for (y of range(m)) x*y)
is
(function* () { for (let x of range(n)) for (let y of range(m)) yield x*y; })()
Desugaring FTW, indeed.
[I guess your "of course" activated my ranting trigger here, given that small core languages plus well-defined desugarings were still regarded best practice last time I looked. ;)]
Yes, small kernel semantics, sugar (macros, if possible) for larger surface language spec, is still tops. Allen may have a thought. ECMA-262 is not set up to facilitate this yet.
On Sep 6, 2013, at 9:04 AM, Brendan Eich wrote:
I keep forgetting how the ES6 draft uses "generator comprehension", not "generator expression" (the latter is a generator function expression). Contrast with Python.
Perhaps "iterator comprehension" would be better terminlogy.
take my tire
Sent through Glass
On Sep 6, 2013, at 9:06 AM, Brendan Eich wrote:
Domenic Denicola <mailto:domenic at domenicdenicola.com>
Why not just make them sugar for creating custom iterable-iterators without shallow coroutine capabilities?
Because we need the shallow continuation to hold the implicitly declared let bindings induced by the for heads.
And we need to handle this binding appropriately.
(for (x of [1,2, 3]) this[x])
can not desugar into
(function*() { for (let x of [1,2, 3]) yield this[x]; })()
because the this binding would be wrong.
Good point, although the example in question did not use this, and I would have adapted if it had:
(function*() { for (let x of [1,2, 3]) yield this[x]; }).call(this)
Too bad that we don't have generator arrow function syntax :-P.
Brendan Eich wrote:
(function*() { for (let x of [1,2, 3]) yield this[x]; }).call(this)
Then of course one has to say "the original value of .call", which is a pain.
Desugaring is less convenient than it ought to be without uniform parameterization support. (Generator arrow functions would help too. :-P^2)
I was originally going to make a comment about this on your blog post, but this now seems like a better place:
In domenic.me/2013/09/06/es6-iterators-generators-and-iterables you state the following difinitions:
A generator is a specific type of iterator which also has a throw method, and whose next method takes a parameter.
A generator function is a special type of function that always returns a generator, and which can use the special contextual keyword yield inside of itself. You can send values or exceptions into the body of the function, at the points where yield appears, via the returned generator's next and throw methods. They are created with function* syntax.
While these definitions aren't "wrong" I think they do not emphasizing the most important points.
I would use the following definitions:
A generator is an iterator whose 'next' results are determined by a single function body or expression. The function body or expression typically includes explicit or implicit 'yield' points which provide the values returned by invocations of the iterator's 'next' method.
A generator function is a function that acts as a factory (or constructor) of a generators with explicit 'yield' points. The body of a generator function is the function body of the generator.
A generator comprehension is an expression that evaluates to a generator with an implicit yield point. The expression within a generator comprehension has an implicit yield and is the expression that provides the values returned by invocations of the iterator's 'next' method.
In other words, the essential difference between generators and the general concept of iterators, it not the presence or absence of the 'throw' method. Instead it is an authoring difference. Using generator functions or generator comprehension, a JS programmer can define an iterator as as if it was a single function or expression rather than an explicit state machine.
Thanks Allen; I agree those are much better definitions and will incorporate them into an edit.
On Sep 6, 2013, at 10:22 AM, Brendan Eich wrote:
Too bad that we don't have generator arrow function syntax :-P.
Almost as good, a arrow function with a generator comprehension as its expression body. For example, a factory for a generator with a lexical this binding:
c => (for (p of c) if (p in this) this[p])
not much different from a hypothetical generator arrow function and arguably better :
c *=> {for (p of c) if (p in this) yield this[p]}
There would be no explicit yield in the second example, though.
(yield is an error in any arrow body.)
I'm not seriously advocating generator arrow function syntax, mind you!
Just want a better way to bind this, e.g.,
strawman:bind_operator.
On Sep 6, 2013, at 11:18 AM, Brendan Eich wrote:
There would be no explicit
yieldin the second example, though. (yieldis an error in any arrow body.)
why no yield? Note that I wrote a { } body for the generator arrow function body, rather than an expression body and that generator functions use explicit |yield| in their bodies.
Also, if the expression body form had an implicit yield in front of the expression (eg, c *=> 42, implicitly yields 42) then then there would be confusion between
c => (for (p of c) if (p in this) this[p]) //function that returns a generator that yields this[p]
c *=> (for (p of c) if (p in this) this[p]) //function that returns a generator that yields a generator
The first form would almost always be what somebody would want.
I'm not seriously advocating generator arrow function syntax, mind you!
Yup, I think we're better off without them.
Allen Wirfs-Brock <mailto:allen at wirfs-brock.com> September 6, 2013 11:48 AM
why no
yield? Note that I wrote a { } body for the generator arrow function body, rather than an expression body and that generator functions use explicityieldin their bodies.
Sorry, brain-fart. You're right.
On that note, if we had generator arrows, why not allow yield in an
expression-body.
On Sep 6, 2013, at 11:56 AM, Brendan Eich wrote:
On that note, if we had generator arrows, why not allow yield in an expression-body.
That would be ok, but without an expression level looping construct, you can't do anything very interesting with it. So that leads back to using a generator comprehension and the sort of confusing stuff in the second part of my previous reply.
On Fri, Sep 6, 2013 at 10:48 AM, Domenic Denicola <domenic at domenicdenicola.com> wrote:
The name works because they're sugar for a generator function immeidately invoked:
I mean, that's true, but why is that true?
It's the simplest way to achieve the desired behavior.
What is the value of allowing you to send in values via
.next(v), or send in exceptions via.throw(e)? Why not just make them sugar for creating custom iterable-iterators without shallow coroutine capabilities?
The coroutine-ness of a generator comprehension seems inherent to me. It creates lexical environments. You can observe this if the generator comprehension creates any closures (or the like) over the variables.
var it = (for (a of A) () => a++);
Other variations on the theme reveal other aspects of the lexical environments; direct eval exposes a bunch more and I think rules out any non-coroutine-based desugaring.
In another, tidier language, perhaps.
So, it's a coroutine. That doesn't mean it must be exposed as a generator. But why not? Having an extra constructor, prototype, and in general an extra concept in the spec for objects that are exactly like generators in every way, but it's maybe not very useful to do certain operations on them, seems like optimizing for the wrong good.
On Fri, Sep 6, 2013 at 10:59 AM, David Bruant <bruant.d at gmail.com> wrote:
The presence of .next and .throw will be unnecessarily confusing to authors.
Well, if you need to explain generator comprehensions to someone, I suggest saying something like:
"It's just shorthand for a generator"
If we changed it, so that generator comprehensions were simply iterators, then you could simply explain:
"It works like a generator, but it gets desugared into an iterator, so there's this one method that you probably won't ever call directly anyway, that isn't there. Oh, what's desugaring? It's when one language feature is explained in terms of others. So for example
(for (x of y) x+1)desugars into something like{_it: y[@@iterator](), next() { var st = this._it.next(); if (!st.done) return {value: st.value + 1, done: false}; return st; }}"
On second thought, maybe equivalences are not so confusing. Maybe gratuitous inconsistencies are confusing.
I don't really see .throw() on a generator-expression as a "footgun" either. It does exactly what the .throw() method does for any user-defined generator, most of which, in practice, will just rethrow the exception—which by happy coincidence is exactly what it says on the label:
gen.throw(exc); // shockingly, this will often throw exc
People will probably be somewhat disoriented the first time they encounter generator comprehensions anyway. Coroutines are a little mind-bending. But throw() really has nothing to do with it. The fact that these expressions produce an object that's exactly the same as another language feature is not an obstacle. I expect it'll help.
Why can't they just return iterable-iterators?
Consider:
const g = (for (x of [1, 2, 3]) x * x); g.next(); // returns { done: false, value: 1 } g.next(5); // ??? g.throw(new Error("boo!")); // ???As far as I can tell from the desugaring at harmony:generator_expressions, the argument to
nextis completely ignored, and doingthrowwill bubble out so thatg.throw(new Error("boo!"))is equivalent tothrow new Error("boo!")except that the former will have some implementation stack frames at the top of the stack trace.What does having these be generators buy us? I am almost sure I'm missing something.
Why can't they just return iterable-iterators? Consider: ```js const g = (for (x of [1, 2, 3]) x * x); g.next(); // returns { done: false, value: 1 } g.next(5); // ??? g.throw(new Error("boo!")); // ??? ``` As far as I can tell from the desugaring at http://wiki.ecmascript.org/doku.php?id=harmony:generator_expressions, the argument to `next` is completely ignored, and doing `throw` will bubble out so that `g.throw(new Error("boo!"))` is equivalent to `throw new Error("boo!")` except that the former will have some implementation stack frames at the top of the stack trace. What does having these be generators buy us? I am almost sure I'm missing something.