yield* desugaring
On Mon 29 Apr 2013 17:37, Andy Wingo <wingo at igalia.com> writes:
let (g = EXPR) { let received = void 0, send = true; while (true) { let next = send ? g.send(received) : g.throw(received); if (next.done) break; try { received = yield next.value; // *** send = true; } catch (e) { received = e; send = false; } }
next.value; }
Beyond the scoping error of "next", this desugaring uses let expressions, which AFAICS are not in the spec and have not been discussed in a few years. Are they actually still a thing?
Andy
Andy Wingo wrote:
Beyond the scoping error of "next", this desugaring uses
letexpressions, which AFAICS are not in the spec and have not been discussed in a few years. Are they actually still a thing?
Just a straw-spec device, not proposed for ES6 or 7.
Was there consensus on the return value of the various generator methods being { value?, done? } for next/send/throw? Is it needed for close?
The desugaring for yield* in the face of using { value?, done? } is more likely (without refutable matching or let expressions for the moment):
let a = yield* EXPR;
let a;
{
let g = EXPR;
let received = void 0, send = true, result = void 0;
try {
while (true) {
let { value, done } = send ? g.send(received) : g.throw(received);
if (done) {
result = value;
break;
}
try {
received = yield value;
send = true;
}
catch (e) {
received = e;
send = false;
}
}
}
finally {
try { g.close(); } catch (ignored) { }
}
a = result;
}
On Mon 29 Apr 2013 19:25, Ron Buckton <rbuckton at chronicles.org> writes:
The desugaring for
yield*in the face of using{ value?, done? }is more likely (without refutable matching orletexpressions for the moment):let a; { [...] a = result; }
Correct me if I am wrong, but I don't think this works, given that like
yield X, yield* X is an expression.
close() does not seem to have much value given that it isn't part of the
iterators specification, and one can do any needed action by doing a
throw() on the iterator and relying on the generator to have a finally
block if needed.
-----Original Message----- From: Andy Wingo [mailto:wingo at igalia.com] Sent: Monday, April 29, 2013 11:56 AM To: Ron Buckton Cc: Brendan Eich; es-discuss Subject: Re: yield* desugaring
On Mon 29 Apr 2013 19:25, Ron Buckton <rbuckton at chronicles.org> writes:
The desugaring for yield* in the face of using { value?, done? } is more likely (without refutable matching or let expressions for the moment):
let a; { [...] a = result; }Correct me if I am wrong, but I don't think this works, given that like "yield X", "yield* X" is an expression.
That's correct, it's not quite right given the fact it needs to be an expression. I've been more focused on how to rewrite this via transpiler in ES5.
Andy Wingo wrote:
close()does not seem to have much value given that it isn't part of the iterators specification, and one can do any needed action by doing athrow()on the iterator and relying on the generator to have a finally block if needed.
But throwing has other unwanted effects, in general. First, you need to
define a generator and give it a name. Then, close callers need to throw
that exception. For this reason, Python defined a built-in exception,
GeneratorExit.
But when we prototyped generators (ES4 days), with Python 2.5 already
done and a good source of design experience to draw from, Igor Bukanov
and I came to the current design that forces a return. A forced return
runs the finallys but of course not catches, and it relieves everyone
from a built-in exception. We killed StopIteration, it would be bad to
add GeneratorExit back.
Igor even took the forced-return solution to python-dev and got a
blessing from Philip J. Eby, as a better solution than GeneratorExit.
On Mon 29 Apr 2013 21:33, Brendan Eich <brendan at mozilla.com> writes:
Andy Wingo wrote:
close() does not seem to have much value given that it isn't part of the iterators specification, and one can do any needed action by doing a throw() on the iterator and relying on the generator to have a finally block if needed.
But throwing has other unwanted effects, in general. First, you need to define a generator and give it a name. Then, close callers need to throw that exception. For this reason, Python defined a built-in exception, GeneratorExit.
In Python, the close() method has some interesting interactions with finally:
>>> def foo():
... try:
... yield 1
... finally:
... yield 2
...
>>> o = foo()
>>> o
<generator object foo at 0x7feb31184f00>
>>> o.__next__()
1
>>> o.close()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: generator ignored GeneratorExit
Incidentally, close() is not the same as forcing a return:
>>> def bar():
... try:
... return 1
... finally:
... yield 2
...
>>> bar().__next__()
2
Python's use case is also different because it specifies that when the generator object is finalized, the close() method gets called -- certainly something we don't want to do for ES6. GvR writes in the 2.5 release notes, when close() was introduced:
The addition of the close() method has one side effect that isn't
obvious. close() is called when a generator is garbage-collected, so
this means the generator's code gets one last chance to run before
the generator is destroyed. This last chance means that
try...finally statements in generators can now be guaranteed to
work; the finally clause will now always get a chance to run. The
syntactic restriction that you couldn't mix yield statements with a
try...finally suite has therefore been removed. This seems like a
minor bit of language trivia, but using generators and try...finally
is actually necessary in order to implement the with statement
described by PEP 343.
http://docs.python.org/release/2.5/whatsnew/pep-342.html
ES6 certainly won't provide a guarantee that finally blocks will run, so it seems to me that the main reason for close() is gone.
I think we should leave the close() method out of the spec.
Andy
Andy Wingo wrote:
Hi Brendan,
On Mon 29 Apr 2013 21:33, Brendan Eich<brendan at mozilla.com> writes:
Andy Wingo wrote:
close() does not seem to have much value given that it isn't part of the iterators specification, and one can do any needed action by doing a throw() on the iterator and relying on the generator to have a finally block if needed. But throwing has other unwanted effects, in general. First, you need to define a generator and give it a name. Then, close callers need to throw that exception. For this reason, Python defined a built-in exception, GeneratorExit.
In Python, the close() method has some interesting interactions with finally:
>>> def foo(): ... try: ... yield 1 ... finally: ... yield 2 ... >>> o = foo() >>> o <generator object foo at 0x7feb31184f00> >>> o.__next__() 1 >>> o.close() Traceback (most recent call last): File "<stdin>", line 1, in<module> RuntimeError: generator ignored GeneratorExit
JS1.7 and up in SpiderMonkey (and I expect Rhino; this was all from ES4):
./Darwin_DBG.OBJ/js js> function foo() { try { yield 1; } finally { yield 2; } } js> o = foo() ({}) js> o.next()
1 js> o.close()
typein:5:0 TypeError: yield from closing generator function foo() { typein:5:0 try { typein:5:0 yield 1; typein:5:0 } finally { typein:5:0 yield 2; typein:5:0 } typein:5:0 }
Incidentally, close() is not the same as forcing a return:
>>> def bar(): ... try: ... return 1 ... finally: ... yield 2 ... >>> bar().__next__() 2
That does not show close in action, though. The generator-iterator runs once there, yielding 2.
Yes, return in try runs finally, but that's independent of close and indeed of generators.
Python's use case is also different because it specifies that when the generator object is finalized, the close() method gets called -- certainly something we don't want to do for ES6.
Quite. Long-ago es4-discuss threads covered all of this.
esdiscuss/2006-December, esdiscuss/2007-January
Look for "Immediate closing of iterators".
ES6 certainly won't provide a guarantee that finally blocks will run, so it seems to me that the main reason for close() is gone.
The JS1.7 prototype for ES4 ran close automatically only from for-in (which evolved into ES6's for-of). That's still on the table.
I think we should leave the close() method out of the spec.
See above. Automating generator close from for-of is straightforward:
js> function foo() { try { yield 1; } finally { print("glorp"); } } js> for (let x of foo()) print(x)
1 glorp
It's useful too, with no obvious alternative when you need it.
On Tue 30 Apr 2013 09:47, Brendan Eich <brendan at mozilla.com> writes:
js> o.close() typein:5:0 TypeError: yield from closing generator function foo() { typein:5:0 try { typein:5:0 yield 1; typein:5:0 } finally { typein:5:0 yield 2; typein:5:0 } typein:5:0 }
For the record, this behavior is not specified in harmony:generators. Also it seems to me to be an unnecessary restriction that originated in Python's desire to always run the finally blocks -- not a goal of ES6.
Incidentally, close() is not the same as forcing a return:
>>> def bar(): ... try: ... return 1 ... finally: ... yield 2 ... >>> bar().__next__() 2That does not show close in action, though. The generator-iterator runs once there, yielding 2.
Sure. I just mean to say that close introduces some mental complication. "A yield expression can produce a value or throw an expression. Except when close() is called, in which case it's like a return. But not quite like a return." The spec would be better without the last two sentences IMO.
Python's use case is also different because it specifies that when the generator object is finalized, the close() method gets called -- certainly something we don't want to do for ES6.
Quite. Long-ago es4-discuss threads covered all of this.
esdiscuss/2006-December, esdiscuss/2007-January
Look for "Immediate closing of iterators".
If a generator runs to completion, there is no need to close it -- the operation is a no-operation.
A bug mentioned in that discussion:
bugzilla.mozilla.org/show_bug.cgi?id=349326
seems to suggest that you expect close() to be called when the generator is collected, which again is not an ES6 thing.
ES6 certainly won't provide a guarantee that finally blocks will run, so it seems to me that the main reason for close() is gone.
The JS1.7 prototype for ES4 ran close automatically only from for-in (which evolved into ES6's for-of). That's still on the table.
Though yield* is currently specified to call close(), I don't believe that close() is in the for-of spec.
See above. Automating generator close from for-of is straightforward:
js> function foo() { try { yield 1; } finally { print("glorp"); } } js> for (let x of foo()) print(x) 1 glorp
This doesn't require close() at all to work! Did you have another example in mind?
It really seems to me that this is an interface that was made because of Python's "always-run-the-finally-block" thing, which ES6 doesn't have. Because we don't have that need, it does not seem useful to me.
Andy
Is the reason why you wouldn't want to run finally blocks in generators
described elsewhere on the list? Choosing not to run a generator's finally
block is, to me at least, a very significant behavioral compromise,
especially for use cases where generators are used to approximate
coroutines (think task.js). Breaking finally would also make it harder to
do resource management for generators that require temporary storage or
other resources.
Any scenarios where a finally block doesn't run are a potential trap for
the unwary, especially because there are so many different ways for a
generator to get manipulated by third-party code. I've personally had bugs
creep into my applications as a result of this sort of thing.
On Tue 30 Apr 2013 10:23, Kevin Gadd <kevin.gadd at gmail.com> writes:
Is the reason why you wouldn't want to run
finallyblocks in generators described elsewhere on the list?
I am not sure I understand the question. Certainly you want run finally blocks in response to normal control flow in a generator:
function* g() {
try {
yield 1;
try { yield 2; } catch (e) { yield e; }
yield 3;
} finally {
yield 4;
}
yield 5;
}
function Sentinel() {}
function Sentinel2() {}
var iter;
iter = g();
assertEquals(1, iter.next());
assertEquals(2, iter.next());
var exn = new Sentinel;
assertEquals(exn, iter.throw(exn));
assertEquals(3, iter.next()); // ***
assertEquals(4, iter.throw(new Sentinel2));
assertThrows(function() { iter.next(); }, Sentinel2);
assertThrows(function() { iter.next(); }, Error);
However. If I understand your question correctly, your question is, if
we just stop after the line marked "***" above, does the finally {yield 4} block ever run?
To run finally blocks when a generator object is suspended within the
try{} of a try/finally block is to introduce finalizers into ECMAScript.
So if this is really what you want, then your question is "why is it a
bad idea to add finalizers to ECMAScript". I think these slides from
Hans Boehm fairly describe the situation:
www.hpl.hp.com/personal/Hans_Boehm/misc_slides/java_finalizers.pdf
Finalizers can be useful but they are not a great interface, they introduce concurrency, and they are otherwise not currently part of the language, and so for example V8's great GC hasn't had to deal with them, which is a win for web performance. Better to avoid adding them to the language, especially in this oblique way.
My apologies. I read your statement "Also it seems to me to be an unnecessary restriction that originated in Python's desire to always run the finally blocks -- not a goal of ES6" as if it were suggesting that ES6 had an existing design philosophy that meant finally blocks would not be reliable in generators. It's fine if a generator's finally block doesn't run if you never terminate the generator (whether through close or through iter.throw or through calling iter.next until it completes).
However, I would definitely expect a given finally block to run if i use for-of or similar on the generator. This is the intent, I hope?
Hi Kevin,
On Tue 30 Apr 2013 11:05, Kevin Gadd <kevin.gadd at gmail.com> writes:
I would definitely expect a given finally block to run if i use for-of or similar on the generator. This is the intent, I hope?
Certainly they run in this situation:
function *g1() { try { yield 1; } finally { qux(); } }
for (x of g1())
print (x)
Or in this one:
function *g2() { try { yield 1; return; } finally { qux(); } }
for (x of g2())
print (x)
But the question is what happens here:
function *g3() { try { yield 1; } finally { qux(); } }
for (x of g3())
break;
Or here:
function *g4() { try { yield 1; } finally { qux(); } }
for (x of g4())
throw "foo";
Or here:
function *g5() { try { yield 1; } finally { qux(); } }
for (x of g5())
call_function_that_throws_an_exception();
For me, it is acceptable in the last three cases to never invoke those
finally blocks. Otherwise, for-of would need to be implicitly
surrounded by a try/finally to manually "close" the generator. It
seems to me that it would have pretty negative perf implications; for
example Crankshaft doesn't currently run on functions with try/finally.
On 30 April 2013 13:30, Andy Wingo <wingo at igalia.com> wrote:
Hi Kevin,
On Tue 30 Apr 2013 11:05, Kevin Gadd <kevin.gadd at gmail.com> writes:
I would definitely expect a given finally block to run if i use for-of or similar on the generator. This is the intent, I hope?
Certainly they run in this situation:
function *g1() { try { yield 1; } finally { qux(); } } for (x of g1()) print (x)
Or in this one:
function *g2() { try { yield 1; return; } finally { qux(); } } for (x of g2()) print (x)
But the question is what happens here:
function *g3() { try { yield 1; } finally { qux(); } } for (x of g3()) break;
Or here:
function *g4() { try { yield 1; } finally { qux(); } } for (x of g4()) throw "foo";
Or here:
function *g5() { try { yield 1; } finally { qux(); } } for (x of g5()) call_function_that_throws_an_exception();
For me, it is acceptable in the last three cases to never invoke those finally blocks. Otherwise, for-of would need to be implicitly surrounded by a try/finally to manually "close" the generator. It seems to me that it would have pretty negative perf implications; for example Crankshaft doesn't currently run on functions with try/finally.
A particular Crankshaft limitation may not be the most convincing argument. :) But clearly, even without that, requiring an implicit finally wrapper for every for-of loop could still be costly -- especially because the last case cannot easily be ruled out at compile time.
I'd also argue that not running the finally blocks of a generator in cases where it is abandoned is consistent with coroutine semantics. In those cases, the generator basically amounts to a coroutine that is still active, but being starved because you never yield back to it again.
Even if we did require a close with for-of loops, the problem would still exist if a generator is run directly through its method interface. There is no way the language can enforce a close in this situation, short of finalization.
The moral is that one should simply avoid putting a yield inside a try-finally. There is no guarantee that control ever returns.
On 30 April 2013 14:19, Andreas Rossberg <rossberg at google.com> wrote:
On 30 April 2013 13:30, Andy Wingo <wingo at igalia.com> wrote:
Hi Kevin,
On Tue 30 Apr 2013 11:05, Kevin Gadd <kevin.gadd at gmail.com> writes:
I would definitely expect a given finally block to run if i use for-of or similar on the generator. This is the intent, I hope?
Certainly they run in this situation:
function *g1() { try { yield 1; } finally { qux(); } } for (x of g1()) print (x)
Or in this one:
function *g2() { try { yield 1; return; } finally { qux(); } } for (x of g2()) print (x)
But the question is what happens here:
function *g3() { try { yield 1; } finally { qux(); } } for (x of g3()) break;
Or here:
function *g4() { try { yield 1; } finally { qux(); } } for (x of g4()) throw "foo";
Or here:
function *g5() { try { yield 1; } finally { qux(); } } for (x of g5()) call_function_that_throws_an_exception();
For me, it is acceptable in the last three cases to never invoke those finally blocks. Otherwise, for-of would need to be implicitly surrounded by a try/finally to manually "close" the generator. It seems to me that it would have pretty negative perf implications; for example Crankshaft doesn't currently run on functions with try/finally.
A particular Crankshaft limitation may not be the most convincing argument. :) But clearly, even without that, requiring an implicit finally wrapper for every for-of loop could still be costly -- especially because the last case cannot easily be ruled out at compile time.
I'd also argue that not running the finally blocks of a generator in cases where it is abandoned is consistent with coroutine semantics. In those cases, the generator basically amounts to a coroutine that is still active, but being starved because you never yield back to it again.
Even if we did require a close with for-of loops, the problem would still exist if a generator is run directly through its method interface. There is no way the language can enforce a close in this situation, short of finalization.
The moral is that one should simply avoid putting a yield inside a try-finally. There is no guarantee that control ever returns.
And as Andy points out correctly, that raises the question whether having 'close' makes much sense at all.
I agree with you except with one nit :)
On Tue 30 Apr 2013 14:19, Andreas Rossberg <rossberg at google.com> writes:
The moral is that one should simply avoid putting a
yieldinside atry-finally. There is no guarantee that control ever returns.
It seems that yield in a try/finally can be useful in a more controlled
environment like within a task.js scheduler, where you know that there
is some entity out there managing coroutine life-cycles.
Andy Wingo wrote:
On Tue 30 Apr 2013 09:47, Brendan Eich<brendan at mozilla.com> writes:
js> o.close() typein:5:0 TypeError: yield from closing generator function foo() { typein:5:0 try { typein:5:0 yield 1; typein:5:0 } finally { typein:5:0 yield 2; typein:5:0 } typein:5:0 }For the record, this behavior is not specified in harmony:generators. Also it seems to me to be an unnecessary restriction that originated in Python's desire to always run the
finallyblocks -- not a goal of ES6.
You're right, this seems a hold-over from six or seven years ago.
Incidentally,
close()is not the same as forcing a return:>>> def bar(): ... try: ... return 1 ... finally: ... yield 2 ... >>> bar().__next__() 2That does not show close in action, though. The generator-iterator runs once there, yielding 2.
Sure. I just mean to say that close introduces some mental complication. "A
yieldexpression can produce a value or throw an expression. Except whenclose()is called, in which case it's like areturn. But not quite like areturn." The spec would be better without the last two sentences IMO.
The generator can't close itself, though, so this is not an issue:
js> function foo() { yield 1; gen.close() }
js> gen = foo()
({})
js> gen.next()
1
js> gen.next()
typein:6:26 TypeError: already executing generator
But not to worry, you've convinced me on other grounds!
Python's use case is also different because it specifies that when the generator object is finalized, the
close()method gets called -- certainly something we don't want to do for ES6. Quite. Long-ago es4-discuss threads covered all of this.esdiscuss/2006-December, esdiscuss/2007-January
Look for "Immediate closing of iterators".
If a generator runs to completion, there is no need to close it -- the operation is a no-operation.
A bug mentioned in that discussion:
bugzilla.mozilla.org/show_bug.cgi?id=349326
seems to suggest that you expect
close()to be called when the generator is collected, which again is not an ES6 thing.
Wow, thanks for the memories! That code is long gone. SpiderMonkey does not do anything from finalization or other GC phases that unprivileged (read: not the debugger) code can observe.
Just so you don't have to deep-dive so vigilantly: everyone agrees we should not expose pre- or post-mortem finalizers -- in JS.
ES6 certainly won't provide a guarantee that finally blocks will run, so it seems to me that the main reason for
close()is gone. The JS1.7 prototype for ES4 rancloseautomatically only fromfor-in(which evolved into ES6'sfor-of). That's still on the table.Though
yield*is currently specified to callclose(), I don't believe thatclose()is in thefor-ofspec.
D'oh. Another gap in the wiki, fixable, but again: you've convinced me.
Easier to drop close.
See above. Automating generator close from
for-ofis straightforward:js> function foo() { try { yield 1; } finally { print("glorp"); } } js> for (let x of foo()) print(x) 1 glorpThis doesn't require
close()at all to work! Did you have another example in mind?
I did, sorry -- forgot the break:
js> for (let x of foo()) {print(x); break}
1
glorp
The idea is to run close inevitable, for all completions of the loop.
This is not the same as finalization. It was on the radar as an ugly
kind of "Python with", but we have try-finally already.
It really seems to me that this is an interface that was made because of Python's "always-run-the-
finally-block" thing, which ES6 doesn't have. Because we don't have that need, it does not seem useful to me.
Agreed. Thanks for pointing this out!
Andreas Rossberg wrote:
And as Andy points out correctly, that raises the question whether having 'close' makes much sense at all.
I think we have evolved away from it. Cc'ing Dave to confirm.
On Apr 30, 2013, at 7:25 AM, Brendan Eich <brendan at mozilla.com> wrote:
Andreas Rossberg wrote:
And as Andy points out correctly, that raises the question whether having 'close' makes much sense at all.
I think we have evolved away from it. Cc'ing Dave to confirm.
I'm cool with it, and I agree on type grounds as well: forcing a return without a value may break the expected return type of the generator, which might not want to return undefined. Whereas the "throws" type makes more sense as an open, extensible type.
On Apr 30, 2013, at 4:55 PM, David Herman <dherman at mozilla.com> wrote:
On Apr 30, 2013, at 7:25 AM, Brendan Eich <brendan at mozilla.com> wrote:
Andreas Rossberg wrote:
And as Andy points out correctly, that raises the question whether having 'close' makes much sense at all.
I think we have evolved away from it. Cc'ing Dave to confirm.
I'm cool with it, and I agree on type grounds as well: forcing a return without a value may break the expected return type of the generator, which might not want to return undefined. Whereas the "throws" type makes more sense as an open, extensible type.
It also has a smell to it: the idea that an expression can cause a return, without the syntactic appearance of return. (I'm not opposed to the idea of being able to return from expressions -- I still love do-expressions. But I prefer return to be a syntactically apparent control effect.)
On Wed 01 May 2013 01:55, David Herman <dherman at mozilla.com> writes:
On Apr 30, 2013, at 7:25 AM, Brendan Eich <brendan at mozilla.com> wrote:
Andreas Rossberg wrote:
And as Andy points out correctly, that raises the question whether having 'close' makes much sense at all.
I think we have evolved away from it. Cc'ing Dave to confirm.
I'm cool with it
Excellent! Thanks all for taking a look at it.
On 1 May 2013 02:06, David Herman <dherman at mozilla.com> wrote:
It also has a smell to it: the idea that an expression can cause a return, without the syntactic appearance of
return. (I'm not opposed to the idea of being able to return from expressions -- I still love do-expressions. But I preferreturnto be a syntactically apparent control effect.)
Of course, there is no need to allow 'return' or friends in do-expressions. In fact, I'd be opposed to that. ;)
On May 2, 2013, at 6:25 AM, Andreas Rossberg <rossberg at google.com> wrote:
On 1 May 2013 02:06, David Herman <dherman at mozilla.com> wrote:
It also has a smell to it: the idea that an expression can cause a return, without the syntactic appearance of
return. (I'm not opposed to the idea of being able to return from expressions -- I still love do-expressions. But I preferreturnto be a syntactically apparent control effect.)Of course, there is no need to allow 'return' or friends in do-expressions. In fact, I'd be opposed to that. ;)
I know, you've already stated that. This is OT.
On May 2, 2013, at 9:51 AM, David Herman <dherman at mozilla.com> wrote:
On May 2, 2013, at 6:25 AM, Andreas Rossberg <rossberg at google.com> wrote:
On 1 May 2013 02:06, David Herman <dherman at mozilla.com> wrote:
It also has a smell to it: the idea that an expression can cause a return, without the syntactic appearance of
return. (I'm not opposed to the idea of being able to return from expressions -- I still love do-expressions. But I preferreturnto be a syntactically apparent control effect.)Of course, there is no need to allow 'return' or friends in do-expressions. In fact, I'd be opposed to that. ;)
I know, you've already stated that. This is OT.
(Sorry to be curt. Just don't need to debate do-expressions here, since they're OT and not on the ES6 agenda.)
Brendan reminded me I never replied to the earlier part of this thread.
On Apr 29, 2013, at 8:37 AM, Andy Wingo <wingo at igalia.com> wrote:
For what it's worth (which is very little) this seems like a decent plan to me.
Actually it's worth a lot to me! Your blog post on StopIteration was part of what got me thinking again about it and prodded me to bring it up with TC39.
The desugaring of "yield* EXPR" with boxed return values would be:
let (g = EXPR) { let received = void 0, send = true; while (true) { let next = send ? g.send(received) : g.throw(received); if (next.done) break; try { received = yield next.value; // *** send = true; } catch (e) { received = e; send = false; } }
next.value; }As you can see on the line marked "***", to desugar to plain "yield" you would have to unbox the value and then allow yield to rebox it. This loses any state on "next" -- there could be other properties set on "next", for example if "g" is a non-generator.
IMHO yield* should be specified to return the result object as-is, without re-boxing. This precludes a straightforward desugaring, but it is probably more flexible.
This is an interesting question. I agree that the intuition of yield* is chaining together continuation frames and therefore "just pass on through" seems like the Right Thing. It's only maybe slightly disconcerting that yield* therefore actually gives you additional power that you can't otherwise express locally (i.e., you have to transform the entire containing generator function). But I'm with you, this seems right to me.
Andy Wingo wrote:
On Wed 01 May 2013 01:55, David Herman<dherman at mozilla.com> writes:
On Apr 30, 2013, at 7:25 AM, Brendan Eich<brendan at mozilla.com> wrote:
Andreas Rossberg wrote:
And as Andy points out correctly, that raises the question whether having 'close' makes much sense at all. I think we have evolved away from it. Cc'ing Dave to confirm. I'm cool with it
Excellent! Thanks all for taking a look at it.
Thank you for poking at this! Once again THE SYSTEM IS WORKING ;-).
Greets,
Thanks for the kind words! And thanks also for the new iteration proposal; it looks great to my ignorant eyes. I'll probably take a look at for-of next.
On Thu 02 May 2013 20:07, David Herman <dherman at mozilla.com> writes:
IMHO yield* should be specified to return the result object as-is, without re-boxing. This precludes a straightforward desugaring, but it is probably more flexible.
This is an interesting question. I agree that the intuition of yield* is chaining together continuation frames and therefore "just pass on through" seems like the Right Thing. It's only maybe slightly disconcerting that yield* therefore actually gives you additional power that you can't otherwise express locally (i.e., you have to transform the entire containing generator function). But I'm with you, this seems right to me.
I was working on this today and thought better of my original proposal. It would be nice to assume that calling next() on a generator returns a { value, done } object, and that is not necessarily the case if we pass on the result from calling some other iterator:
let next = send ? g.send(received) : g.throw(received);
Here "g" might not be a generator object -- though we could specify that it is [*] -- and "next" might not have the { value, done } form. It might not even be an object, in which case getting the "value" would fail at an even greater distance from the original error.
This is a very small point in the scheme of things, but it seemed to me that desugaring as "yield next.value" was probably less confusing to users. WDYT?
,
Andy
[*] If we specify that "g" is a generator object, then this question is moot: the identity of the result of "g.send(receiver)" is not visible.
I literally just finished making the changes to the ES6 specification draft to fully incorporate generator semantics. I'll will be making that draft available within the next day or two after worked down the bug backlog a bit.
Because this thread started while I was in the middle of that work, I choose to ignore it until I had finished with it (with the exception, that I differed implementing <generator>.close() because I did notice that its utility was being questioned). I've now digested the thread and I want to provide some feedback on it that reflects my spec. work.
First, as a general comment, I don't use direct desugaring within the spec. but instead use the spec. pseudo code formalisms. This gives me direct access to mechanisms such as internal Completion values and allows me to express behaviors that are difficult or impossible to express via desugarings.
As now specified:
-
I specified
yield*such that it will work with any iterator, not just generators. To me, this seems essential. Otherwise client code is sensitive to other peoples implementation decision (that are subject to change) regarding whether to use a generator or an object based iterator. This was easy to accomplish and only requires a one-time behavioral check to determine whethernextorsendshould be used to retrieve values from the delegated iterator and an behavior guard on invokingthrowon the delegated iterator. -
yield*invokes the @@iterator method on its expression to obtain the iterator. This means you can say things like:yield * [2,4,6,8,10]; //individually yield positive integers <= 10. -
yield*yields thenextResultobject produced by the inner iterator. No unwrapping/rewrapping required. -
I haven't (yet) provided a
closemethod for generators. I still think we should. Unwinding via return is clearly the appropriate semantics forclose. Contrary to some concerns expressed on this thread, I don't think there are any issues with distinguishingclosetriggered returns from actual user level concerns. Such returns (all returns) have to pass as Completion values through the outer body level of the generator and it is easy enough (at the spec level) for me to tag such returns (using a unique, unobservable internal return value) such that there is no confusion with an actual user level return.I think
closeis an useful and important operation for maintaining finally semantics in two use cases:-
Â
for (v of iterable) {...break;...} /* or return or throw or outer continue*/The
for-ofshould should automatically invokecloseif the loop is terminated before exhausting the iterator. Thecloseneeds to be guarded with a behavioral check just like I did foryield*. I think this is a common case and we really should be doing our best effort to maintain the reliability offinallyblocks for this common case. -
Any time user code is manually draining a known generator that it opened and decides that it is now done with the generator. They really should close it. Of course, they may not, but regardless they should be provided with a means to do so and it should be encouraged as a best practice.
The second use case is perhaps a little iffy, but the first really seems fundamental to finally semantics and something that the language can automatically do. To me, it would seem a bit negligent to not handle that situation.
-
On Sun 12 May 2013 21:29, Allen Wirfs-Brock <allen at wirfs-brock.com> writes:
- I specified
yield*such that it will work with any iterator, not just generators. To me, this seems essential. Otherwise client code is sensitive to other peoples implementation decision (that are subject to change) regarding whether to use a generator or an object based iterator. This was easy to accomplish and only requires a one-time behavioral check to determine whethernextorsendshould be used to retrieve values from the delegated iterator and an behavior guard on invokingthrowon the delegated iterator.
Are you checking structurally or are you checking the internal "brand"? I can think of situations where you would want to decorate a generator iterator, producing an object with the same interface but not actually a generator iterator. Perhaps one could decorate with another generator, though. Relatedly:
yield*invokes the @@iterator method on its expression to obtain the iterator. This means you can say things like:yield * [2,4,6,8,10]; //individually yield positive integers <= 10.
Is it wise to do this? It would also be possible to define
function* iterate(iterable) { for (let x of iterable) yield x; }
and that would provide a uniform interface to the RHS of a yield*, and a
natural point at which a throw() to a generator suspended in a yield*
would raise an exception.
yield*yields thenextResultobject produced by the inner iterator. No unwrapping/rewrapping required.
Does this ensure that the result is an object and has value and done
properties?
- I haven't (yet) provided a
closemethod for generators. I still think we should.
Let me try to summarize some thoughts on close(). I'll start from one
of your use cases.
b) Any time user code is manually draining a known generator that it opened and decides that it is now done with the generator. They really should close it. Of course, they may not, but regardless they should be provided with a means to do so and it should be encouraged as a best practice.
I think the question to ask is, why do you think this is a good
recommendation? It can't be for general resource cleanup issues,
because otherwise iterators would also have a close method. So I am
inclined to think that it is because you see finally in the source
code, and you treat that as a contract with the user that a finally
block actually does run for its effects.
But that's precisely what we can't guarantee: unlike function
activations, the dynamic extent of a generator activation is unlimited.
We don't have finalizers, so we can't ensure that a finally block runs.
And once we allow for that possibility, it seems to me that close() is
not only less useful, but that by making a kind of promise that we can't
keep, it can be harmful.
close() also complicates a user's mental model of what happens when they
see a yield. I see yield x and I think, OK, this suspends
computation. If it's in a position to yield a value, that tells me that
it might also produce a value. That it could throw an exception is less
apparent, so you have to remember that. But then you also have to
remember that it might be like a return! It's that second invisible
behavior that tips my mental balance.
Of course a close() doesn't actually force a generator activation to
finish; there is the possibility of exceptions or further yields in
finally blocks. In this case Python will re-queue the generator for
closing, so you do finally run all the finallies -- but again, we don't
guarantee that, so add that to the mental model of what close does...
close() also adds restrictions on the use of generator objects. For
example it's very common to use a loop variable after a loop:
for (i = 0; i < N; i++) {
...
if (foo) break;
}
...
// here we use i
One can imagine situations in which it would be nice to use a generator object after a break, for example in lazy streams:
function* fib() {
var x = 0, y = 1;
yield x;
yield y;
while (1) {
let z = x + y;
yield x + y;
x = y;
y = z;
}
}
function for_each_n(f, iter, n) {
if (n) {
for (let x of iter) {
f(x);
if (--n == 0)
break;
}
}
}
var iter = fib();
for_each_n(x => console.log(x), iter, 10); // first 10
for_each_n(x => console.log(x), iter, 10); // next 10
In summary my problems with close() are these:
-
It attempts to provide a reliable finally for unlimited-extent activations, when we can't do that.
-
It complicates the mental model of what happens when you yield.
-
It makes common sugar like for-of inappropriate for some uses of generator objects.
WDYT? Not to distract you too much from the new draft, of course :)
On 12 May 2013 21:29, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
First, as a general comment, I don't use direct desugaring within the spec. but instead use the spec. pseudo code formalisms. This gives me direct access to mechanisms such as internal Completion values and allows me to express behaviors that are difficult or impossible to express via desugarings.
I could say a few things about why I think this is not actually a good approach in general, but that's another discussion...
As now specified:
I specified yield* such that it will work with any iterator, not just generators. To me, this seems essential. Otherwise client code is sensitive to other peoples implementation decision (that are subject to change) regarding whether to use a generator or an object based iterator. This was easy to accomplish and only requires a one-time behavioral check to determine whether "next" or "send" should be used to retrieve values from the delegated iterator and an behavior guard on invoking "throw" on the delegated iterator.
yield* invokes the @@iterator method on its expression to obtain the iterator. This means you can say things like: yield * [2,4,6,8,10]; //individually yield positive integers <= 10.
Is that a good idea? I'd be fine with it if you could transparently generalise to iterators, but you can't. You need to special-case iterators that are generators, and for them the semantics will be quite different. For example, they will recursively handle sends and throws to the outer generator, whereas for other iterators, where will those even go? In a nutshell, what you are suggesting is to break the iterator abstraction.
In any case, I think this is a substantial enough change from the proposal that it needs consensus first.
yield* yields the nextResult object produced by the inner iterator. No unwrapping/rewrapping required.
I haven't (yet) provided a "close" method for generators. I still think we should. Unwinding via return is clearly the appropriate semantics for "close". Contrary to some concerns expressed on this thread, I don't think there are any issues with distinguishing "close" triggered returns from actual user level concerns. Such returns (all returns) have to pass as Completion values through the outer body level of the generator and it is easy enough (at the spec level) for me to tag such returns (using a unique, unobservable internal return value) such that there is no confusion with an actual user level return.
I think "close" is an useful and important operation for maintaining finally semantics in two use cases: a) for (v of iterable) {...break;...} /* or return or throw or outer continue*/ The for-of should should automatically invoke "close" if the loop is terminated before exhausting the iterator. The "close" needs to be guarded with a behavioral check just like I did for yield*. I think this is a common case and we really should be doing our best effort to maintain the reliability of finally blocks for this common case.
A couple of observations:
-
This again seems to require breaking the iterator abstraction, since you have to special-case the described close behaviour for iterators that are generators (you cannot close other iterators).
-
It amounts to requiring every for-of loop over a generator to be wrapped into an implicit try-statement. That is doable, but imposes a substantial cost, which we had just removed by retiring StopIteration.
-
Whatever way you turn it, close is not a well-behaved API, because (a) we cannot guarantee it being called, and (b) a generator could actually intercept it in a try-finally and re-yield another result (same problem as with the StopIteration exception before). What would that mean?
On May 13, 2013, at 1:22 AM, Andy Wingo wrote:
Hi,
On Sun 12 May 2013 21:29, Allen Wirfs-Brock <allen at wirfs-brock.com> writes:
- I specified yield* such that it will work with any iterator, not just generators. To me, this seems essential. Otherwise client code is sensitive to other peoples implementation decision (that are subject to change) regarding whether to use a generator or an object based iterator. This was easy to accomplish and only requires a one-time behavioral check to determine whether "next" or "send" should be used to retrieve values from the delegated iterator and an behavior guard on invoking "throw" on the delegated iterator.
Are you checking structurally or are you checking the internal "brand"? I can think of situations where you would want to decorate a generator iterator, producing an object with the same interface but not actually a generator iterator. Perhaps one could decorate with another generator, though. Relatedly:
Structurally, of course. ES is a dynamic language. yield* has no dependencies upon branded implementation details of the object it is delegating to. Even without the very minor accommodations I made for the differences between the Iterator and generator interfaces users, with the Wiki desugaring users could accomplishing the same thing simply by providing a "send" method (and "throw" for completeness") on the iterator they are delegating to. They might even write themselves an ItrAsGen utility function.
BTW, this is only an issue because Iterator and generators interfaces are more different then they need to be. More latter...
- yield* invokes the @@iterator method on its expression to obtain the iterator. This means you can say things like: yield * [2,4,6,8,10]; //individually yield positive integers <= 10.
Is it wise to do this?
Every other place in the language (syntax and built-in functions) where we expect Iterables we transparently accept either iterator or generator objects. Why should yield* be any different. The generator wiki proposal even describes yield* as similar to to a for loop over the generator. This is how regular uses are going to think about it, so yes it is wise.
It would also be possible to define
function* iterate(iterable) { for (let x of iterable) yield x; }
and that would provide a uniform interface to the RHS of a yield*, and a natural point at which a throw() to a generator suspended in a yield* would raise an exception.
In fact it "is" possible to define this and it is one way to implement the ItrAsGen I mention above. Or you could just write: yield * (for (Iet x of iterable) x);
But why force uses of yield * to defensive program in that way and why force the runtime overhead of a second level of generator iteration when (those context swaps must cost something) if in fact a simple object based iterator is being used.
- yield* yields the nextResult object produced by the inner iterator. No unwrapping/rewrapping required.
Does this ensure that the result is an object and has "value" and "done" properties?
Of course not, that wouldn't that would be an unnecessary extra runtime type check. ES is a dynamic language and there is no guarantee that an iterator actually provides such an object. However, if your nextResult object doesn't have or inherit those properties their value is a falsy undefined, so "done" is false. BTW, this is another concern I have about the current nextResult object design. I would prefer that a missing "done" property mean the same thing as done: true.
- I haven't (yet) provided a "close" method for generators. I still think we should.
Let me try to summarize some thoughts on close(). I'll start from one of your use cases.
b) Any time user code is manually draining a known generator thatit opened and decides that it is now done with the generator. They really should close it. Of course, they may not, but regardless they should be provided with a means to do so and it should be encouraged as a best practice.
I think the question to ask is, why do you think this is a good recommendation? It can't be for general resource cleanup issues, because otherwise iterators would also have a close method. So I am inclined to think that it is because you see "finally" in the source code, and you treat that as a contract with the user that a finally block actually does run for its effects.
It's all about "finally" which up to now has been a strong guarantee.
But that's precisely what we can't guarantee: unlike function activations, the dynamic extent of a generator activation is unlimited. We don't have finalizers, so we can't ensure that a finally block runs. And once we allow for that possibility, it seems to me that close() is not only less useful, but that by making a kind of promise that we can't keep, it can be harmful.
The most common uses of generators (for-of, and similar contexts ) could fulfill the "finally" guarantees, even in the presence of early exits. Manual draining of generators ( case b) is the exceptional case. Providing "close" still doesn't guarantee the integrity of "finally" for that case but it does provide the necessary mechanism for building generator consuming abstractions that can make that guarantee.
****** most important point:
However, there is another alternative to close that addresses the finally issue. I can make it statically illegal for a "yield" statement to be nested within a block that is protected by a "finally". This preserve the integrity of "finally" which is my only concern. It also address the for-of early exit issues. This restriction is trivial to specify (and should be only slightly less trivial to implement). Since I don't currently have "close" in the spec. I probably should have done this anyway.
close() also complicates a user's mental model of what happens when they see a "yield". I see "yield x" and I think, OK, this suspends computation. If it's in a position to yield a value, that tells me that it might also produce a value. That it could throw an exception is less apparent, so you have to remember that. But then you also have to remember that it might be like a "return"! It's that second invisible behavior that tips my mental balance.
Note it really isn't like return from the user's perspecitve, it is more like a "break" out of the generator. That's a better way to think about it.
Of course a close() doesn't actually force a generator activation to finish; there is the possibility of exceptions or further yields in finally blocks. In this case Python will re-queue the generator for closing, so you do finally run all the finallies -- but again, we don't guarantee that, so add that to the mental model of what "close" does...
close() also adds restrictions on the use of generator objects. For example it's very common to use a loop variable after a loop:
for (i = 0; i < N; i++) { ... if (foo) break; } ... // here we use i
One can imagine situations in which it would be nice to use a generator object after a break, for example in lazy streams:
function* fib() { var x = 0, y = 1; yield x; yield y; while (1) { let z = x + y; yield x + y; x = y; y = z; } } function for_each_n(f, iter, n) { if (n) { for (let x of iter) { f(x); if (--n == 0) break; } } } var iter = fib(); for_each_n(x => console.log(x), iter, 10); // first 10 for_each_n(x => console.log(x), iter, 10); // next 10
In summary my problems with close() are these:
(1) It attempts to provide a reliable finally for unlimited-extent activations, when we can't do that.
(2) It complicates the mental model of what happens when you yield.
(3) It makes common sugar like for-of inappropriate for some uses of generator objects.
I think the static occurrence of "yield" within the control of a "finally" is the simplest solution. Thanks for helping me understand that.
On May 13, 2013, at 2:07 AM, Andreas Rossberg wrote:
On 12 May 2013 21:29, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
First, as a general comment, I don't use direct desugaring within the spec. but instead use the spec. pseudo code formalisms. This gives me direct access to mechanisms such as internal Completion values and allows me to express behaviors that are difficult or impossible to express via desugarings.
I could say a few things about why I think this is not actually a good approach in general, but that's another discussion...
As now specified:
I specified yield* such that it will work with any iterator, not just generators. To me, this seems essential. Otherwise client code is sensitive to other peoples implementation decision (that are subject to change) regarding whether to use a generator or an object based iterator. This was easy to accomplish and only requires a one-time behavioral check to determine whether "next" or "send" should be used to retrieve values from the delegated iterator and an behavior guard on invoking "throw" on the delegated iterator.
yield* invokes the @@iterator method on its expression to obtain the iterator. This means you can say things like: yield * [2,4,6,8,10]; //individually yield positive integers <= 10.
Is that a good idea? I'd be fine with it if you could transparently generalise to iterators, but you can't. You need to special-case iterators that are generators, and for them the semantics will be quite different. For example, they will recursively handle sends and throws to the outer generator, whereas for other iterators, where will those even go? In a nutshell, what you are suggesting is to break the iterator abstraction.
Yes, it's a very good idea. The easy way for an imperative programmer (there are a few of us in the world) to understand yield* is as a yielding loop over an iterator. Very straight forward to understand rather than describing it as a "mechanism for composing generators" (from the wiki) which I had no idea what it meant until I carefully studied the desugaring. At that point it because clear that it was just a yielding loop over an iterator that for some reason was arbitrarily being restricted to being a generator.
That restriction is what is breaking the iterator abstraction. The most common use case of generators is for implementing iterators. If I'm implementing an iterator via a generator and I have to perform a inner-iteration over an contained iterable (for example, some sort of flattening operation) the way I code that inner iteration shouldn't depend upon whether or not the implementor of the inner iterable chose to use a generator rather than a stateful object as the iterator. If that is the case, then I am essentially precluded from using yield* in this most common situation. Instead I could only use yield* in situations where I know that the target object is implemented via a generator. To me, that is to gross violation of the iteration abstraction and would call into question why we even have yield*.
Could you clarify the special-case handling you have in mind? There is nothing in the wiki proposal desugaring of yield* that guarantees that the delegated object is an actual generator. All it requires is a "send" method (and, in that desugaring, "close" plus "throw" if "throw" is actually invoked). Once a "send" is invoked (whatever it does) you're operating within the delegated object and it doesn't (and shouldn't) know if it was invoked as part of a yield* or a for-of or an explicit method call.
Regarding recursive "sends" to an outer generator. This shouldn't work, according to the wiki proposal. When executing a yield* the outer generator must be in the "executing" state. Invoking an inner generator from an yield* via a "send" invocation still leaves the outer generator in the "executing" state. If the inner generator invokes "send" on the outer generator the "send" will throw because the outer is already in the "executing" state.
Our real problem here seems to be that generators specialize the iterator abstraction in ways that make it hard to use them interchangeably.
First why do we need "send" at all. Why not simply allow an argument to be passed to "next" (of course, it is already allowed) and leave it up to the generator implementation as to whether or not they pay any attention to it. Clearly a client needs to be aware when they are using a generator that expects to receive a value back from yield so that fact must be documented in the public contract of that generator. Once that is done, the client can use "next" as easily as they could use "send". Of course, if people really like the name "send" we could also provide that method for generators with the meaning: send(value) {return this.next(value)}
That leaves only "throw" as an issue. Personally, I'd just make it part of the Iterator interface and provide an Iterator abstract class that provides throw(exception) {throw exception} as the default "throw" implementation so most iterator authors don't even have to think about it. Short of that, I think having an explicit behavior check for "throw" in the yield* algorithm is a very small cost (that only arises if someone actually invokes the "throw" method on the outer generator) and would take care of most common situation where "throw" is likely to be invoked on an iterator..
In any case, I think this is a substantial enough change from the proposal that it needs consensus first.
That's why I brought it to attention here...
yield* yields the nextResult object produced by the inner iterator. No unwrapping/rewrapping required.
I haven't (yet) provided a "close" method for generators. I still think we should. Unwinding via return is clearly the appropriate semantics for "close". Contrary to some concerns expressed on this thread, I don't think there are any issues with distinguishing "close" triggered returns from actual user level concerns. Such returns (all returns) have to pass as Completion values through the outer body level of the generator and it is easy enough (at the spec level) for me to tag such returns (using a unique, unobservable internal return value) such that there is no confusion with an actual user level return.
I think "close" is an useful and important operation for maintaining finally semantics in two use cases: a) for (v of iterable) {...break;...} /* or return or throw or outer continue*/ The for-of should should automatically invoke "close" if the loop is terminated before exhausting the iterator. The "close" needs to be guarded with a behavioral check just like I did for yield*. I think this is a common case and we really should be doing our best effort to maintain the reliability of finally blocks for this common case.
A couple of observations:
In my response to Andy I concluded that syntactically restricting yield to not be finally protected is the better solution.
On 13 May 2013 17:58, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
Every other place in the language (syntax and built-in functions) where we expect Iterables we transparently accept either iterator or generator objects. Why should yield* be any different.
Generators are subtypes of iterators, but not the other way round. So it is to be expected that there are contexts where you can use the latter but not the former. And 'yield' is quite naturally a feature of generators.
However, there is another alternative to close that addresses the finally issue. I can make it statically illegal for a "yield" statement to be nested within a block that is protected by a "finally". This preserve the integrity of "finally" which is my only concern. It also address the for-of early exit issues. This restriction is trivial to specify (and should be only slightly less trivial to implement). Since I don't currently have "close" in the spec. I probably should have done this anyway.
Yes, we had the same discussion in the V8 team the other week. It's a reasonable restriction, but it's also an ad-hoc restriction, which I'm personally not overly fond of. Moreover, there might be use cases where you control both definition and consumption of a generator, so that using finally might be safe and sensible. (On the other hand, there aren't really many meaningful use cases for finally to start with, so I don't worry too much about that.)
In short, I could live with that, but don't think it's necessary.
On Mon 13 May 2013 19:24, Allen Wirfs-Brock <allen at wirfs-brock.com> writes:
[
yield*is] just a yielding loop over an iterator that for some reason was arbitrarily being restricted to being a generator.
I don't think the restriction you mention was present in the harmony document, was it?
For what it's worth:
If I'm implementing an iterator via a generator and I have to perform a inner-iteration over an contained iterable (for example, some sort of flattening operation) the way I code that inner iteration shouldn't depend upon whether or not the implementor of the inner iterable chose to use a generator rather than a stateful object as the iterator.
+1
First why do we need
sendat all. Why not simply allow an argument to be passed tonext(of course, it is already allowed) and leave it up to the generator implementation as to whether or not they pay any attention to it.
+1
That leaves only
throwas an issue. Personally, I'd just make it part of the Iterator interface and provide an Iterator abstract class that providesthrow(exception) {throw exception}as the default
throwimplementation so most iterator authors don't even have to think about it.
+1
On May 13, 2013, at 10:42 AM, Andreas Rossberg wrote:
On 13 May 2013 17:58, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
Every other place in the language (syntax and built-in functions) where we expect Iterables we transparently accept either iterator or generator objects. Why should yield* be any different.
Generators are subtypes of iterators, but not the other way round. So it is to be expected that there are contexts where you can use the latter but not the former. And 'yield' is quite naturally a feature of generators.
But we are talking about yield* rather than yield. It is completely irrelevant to yield* whether the inner iterator implements next/send via an direct method return or an encapsulated yield.
However, there is another alternative to close that addresses the finally issue. I can make it statically illegal for a
yieldstatement to be nested within a block that is protected by afinally. This preserve the integrity offinallywhich is my only concern. It also address thefor-ofearly exit issues. This restriction is trivial to specify (and should be only slightly less trivial to implement). Since I don't currently haveclosein the spec. I probably should have done this anyway.Yes, we had the same discussion in the V8 team the other week. It's a reasonable restriction, but it's also an ad-hoc restriction, which I'm personally not overly fond of. Moreover, there might be use cases where you control both definition and consumption of a generator, so that using finally might be safe and sensible. (On the other hand, there aren't really many meaningful use cases for finally to start with, so I don't worry too much about that.)
In short, I could live with that, but don't think it's necessary.
It closes down this whole edge-case focused discussion and that's valuable in itself. Also, since it turns try {yield expr} finally{} into a syntax error we could revisit the decision in a future edition if somebody actually comes up with compelling use cases.
I've alway made this trivial change to my working spec. draft. If consensus prevails against it, its easy enough to undo, but for now I suggest we move on with other issues.
On 13 May 2013 19:24, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
On May 13, 2013, at 2:07 AM, Andreas Rossberg wrote:
On 12 May 2013 21:29, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
As now specified:
I specified yield* such that it will work with any iterator, not just generators. To me, this seems essential. Otherwise client code is sensitive to other peoples implementation decision (that are subject to change) regarding whether to use a generator or an object based iterator. This was easy to accomplish and only requires a one-time behavioral check to determine whether "next" or "send" should be used to retrieve values from the delegated iterator and an behavior guard on invoking "throw" on the delegated iterator.
yield* invokes the @@iterator method on its expression to obtain the iterator. This means you can say things like: yield * [2,4,6,8,10]; //individually yield positive integers <= 10.
Is that a good idea? I'd be fine with it if you could transparently generalise to iterators, but you can't. You need to special-case iterators that are generators, and for them the semantics will be quite different. For example, they will recursively handle sends and throws to the outer generator, whereas for other iterators, where will those even go? In a nutshell, what you are suggesting is to break the iterator abstraction.
Yes, it's a very good idea. The easy way for an imperative programmer (there are a few of us in the world) to understand yield* is as a yielding loop over an iterator. Very straight forward to understand rather than describing it as a "mechanism for composing generators" (from the wiki) which I had no idea what it meant until I carefully studied the desugaring. At that point it because clear that it was just a yielding loop over an iterator that for some reason was arbitrarily being restricted to being a generator.
That restriction is what is breaking the iterator abstraction.
See my previous reply. I think there is some confusion here about the direction of the abstraction.
Could you clarify the special-case handling you have in mind? There is nothing in the wiki proposal desugaring of yield* that guarantees that the delegated object is an actual generator. All it requires is a "send" method (and, in that desugaring, "close" plus "throw" if "throw" is actually invoked).
Exactly. But iterators don't currently have 'send' and 'throw' methods. So you would want to do something different for generator-like objects then you'd do for other iterators.
Regarding recursive "sends" to an outer generator. This shouldn't work, according to the wiki proposal. When executing a yield* the outer generator must be in the "executing" state. Invoking an inner generator from an yield* via a "send" invocation still leaves the outer generator in the "executing" state. If the inner generator invokes "send" on the outer generator the "send" will throw because the outer is already in the "executing" state.
The case I was talking about is simply this:
function* g() { yield* [1, 2] }
var o = g() o.send(undefined) o.send(5) // what does this mean?
But I suppose the answer is that the sent value is just dropped on the floor, as per the iterator expression interpretation you gave in the other post. Makes sense, I guess.
First why do we need "send" at all. Why not simply allow an argument to be passed to "next" (of course, it is already allowed) and leave it up to the generator implementation as to whether or not they pay any attention to it. Clearly a client needs to be aware when they are using a generator that expects to receive a value back from yield so that fact must be documented in the public contract of that generator. Once that is done, the client can use "next" as easily as they could use "send". Of course, if people really like the name "send" we could also provide that method for generators with the meaning: send(value) {return this.next(value)}
I happen to dislike the name 'send' a lot and would rather call it 'resume'. ;) But your suggestion of merging it with 'next' sounds plausible as well.
That leaves only "throw" as an issue. Personally, I'd just make it part of the Iterator interface and provide an Iterator abstract class that provides throw(exception) {throw exception} as the default "throw" implementation so most iterator authors don't even have to think about it. Short of that, I think having an explicit behavior check for "throw" in the yield* algorithm is a very small cost (that only arises if someone actually invokes the "throw" method on the outer generator) and would take care of most common situation where "throw" is likely to be invoked on an iterator..
That might actually work. If we manage to truly unify iterator and generator types then you got me convinced.
On May 13, 2013, at 10:51 AM, Andy Wingo wrote:
On Mon 13 May 2013 19:24, Allen Wirfs-Brock <allen at wirfs-brock.com> writes:
[
yield*is] just a yielding loop over an iterator that for some reason was arbitrarily being restricted to being a generator.I don't think the restriction you mention was present in the harmony document, was it?
Not precisely, but its unconditional use of send/throw/close in that proposal precluded using any simple (only implements next) iterator, including all of the built-in iterators.
On Mon, May 13, 2013 at 11:08 AM, Andreas Rossberg <rossberg at google.com> wrote:
The case I was talking about is simply this:
function* g() { yield* [1, 2] } var o = g() o.send(undefined) o.send(5) // what does this mean?But I suppose the answer is that the sent value is just dropped on the floor, as per the iterator expression interpretation you gave in the other post. Makes sense, I guess.
Yes, because that code is equivalent to:
function* g() {
yield 1;
yield 2;
}
Using .send() instead of .next() makes no difference to this code.
Using .throw() should just percolate the error up to the yield*
expression, as Allen suggested.
On May 13, 2013, at 11:08 AM, Andreas Rossberg wrote:
On 13 May 2013 19:24, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
On May 13, 2013, at 2:07 AM, Andreas Rossberg wrote: [...] Yes, it's a very good idea. The easy way for an imperative programmer (there are a few of us in the world) to understand yield* is as a yielding loop over an iterator. Very straight forward to understand rather than describing it as a "mechanism for composing generators" (from the wiki) which I had no idea what it meant until I carefully studied the desugaring. At that point it because clear that it was just a yielding loop over an iterator that for some reason was arbitrarily being restricted to being a generator.
That restriction is what is breaking the iterator abstraction.
See my previous reply. I think there is some confusion here about the direction of the abstraction.
Likewise, but I won't argue it again as I think we are converging upon a fine solution.
Could you clarify the special-case handling you have in mind? There is nothing in the wiki proposal desugaring of yield* that guarantees that the delegated object is an actual generator. All it requires is a "send" method (and, in that desugaring, "close" plus "throw" if "throw" is actually invoked).
Exactly. But iterators don't currently have 'send' and 'throw' methods. So you would want to do something different for generator-like objects then you'd do for other iterators.
Or align the interfaces so that the differences don't exist for the situations when the two abstractions are reasonable alternatives.
Regarding recursive "sends" to an outer generator. This shouldn't work, according to the wiki proposal. When executing a yield* the outer generator must be in the "executing" state. Invoking an inner generator from an yield* via a "send" invocation still leaves the outer generator in the "executing" state. If the inner generator invokes "send" on the outer generator the "send" will throw because the outer is already in the "executing" state.
The case I was talking about is simply this:
function* g() { yield* [1, 2] }
var o = g() o.send(undefined) o.send(5) // what does this mean?
But I suppose the answer is that the sent value is just dropped on the floor, as per the iterator expression interpretation you gave in the other post. Makes sense, I guess.
Right. whenever you "send" a value back to a generator you are doing something that requires specific and specialized understanding of the "receiver". If you are sending to an arbitrary generator/iterator you should have no expectation other than the the value is likely to be dropped on the floor,.
First why do we need "send" at all. Why not simply allow an argument to be passed to "next" (of course, it is already allowed) and leave it up to the generator implementation as to whether or not they pay any attention to it. Clearly a client needs to be aware when they are using a generator that expects to receive a value back from yield so that fact must be documented in the public contract of that generator. Once that is done, the client can use "next" as easily as they could use "send". Of course, if people really like the name "send" we could also provide that method for generators with the meaning: send(value) {return this.next(value)}
I happen to dislike the name 'send' a lot and would rather call it 'resume'. ;) But your suggestion of merging it with 'next' sounds plausible as well.
+1 (myself, really :-)
That leaves only "throw" as an issue. Personally, I'd just make it part of the Iterator interface and provide an Iterator abstract class that provides throw(exception) {throw exception} as the default "throw" implementation so most iterator authors don't even have to think about it. Short of that, I think having an explicit behavior check for "throw" in the yield* algorithm is a very small cost (that only arises if someone actually invokes the "throw" method on the outer generator) and would take care of most common situation where "throw" is likely to be invoked on an iterator..
That might actually work. If we manage to truly unify iterator and generator types then you got me convinced.
Which? The abstract class or the guarded throw in yield* (or both).
I think if we can unify all of this we will have simplified the language in a way that will benefit many users for a long time to come.
Andy Wingo wrote:
But that's precisely what we can't guarantee: unlike function activations, the dynamic extent of a generator activation is unlimited. We don't have finalizers, so we can't ensure that a finally block runs.
Otherwise browsers would be even more trivially DoS-attacked.
We went over this in 2006 and agreed then that finally cannot be guaranteed. This is true even without generators:
function foo() { try { while (true); } finally { alert('so sorry'); } } foo()
Browsers must police infinte loops and offer to kill such runaways. The finally can't run.
On May 13, 2013, at 10:51 AM, Andy Wingo <wingo at igalia.com> wrote:
If I'm implementing an iterator via a generator and I have to perform a inner-iteration over an contained iterable (for example, some sort of flattening operation) the way I code that inner iteration shouldn't depend upon whether or not the implementor of the inner iterable chose to use a generator rather than a stateful object as the iterator.
+1
Agree with the goal, as long as we can sensible achieve it.
First why do we need "send" at all. Why not simply allow an argument to be passed to "next" (of course, it is already allowed) and leave it up to the generator implementation as to whether or not they pay any attention to it.
+1
I'm fine with that; also fine with Andreas's "resume" name, which would help with learnability if nothing else. (I have found I've had the most success introducing beginners to generator functions by describing them as being like regular functions that you can "pause.") Of course, the "next" name is more appropriate for the for-of use case. (I'm laying down my bikeshed brush on this one.)
That leaves only "throw" as an issue. Personally, I'd just make it part of the Iterator interface and provide an Iterator abstract class that provides throw(exception) {throw exception} as the default "throw" implementation so most iterator authors don't even have to think about it.
+1
I aaaalmost agree. I want to agree. But part of the importance of the structural type is that you don't have to use subclassing. You can't "mixin" this default throw-through behavior with a base class. We could, of course, offer a mixin function:
iterator({ next: ... })
But that sucks given that we expect most iterators will want the default behavior. Alternatively, we could make the .throw method optional and have it automatically throw-through. I was worried at first that that might be creepy, but really it's kind of sensible. It's an optional iterator method for the MOP that has a sensible default behavior.
On May 13, 2013, at 11:07 AM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
It closes down this whole edge-case focused discussion and that's valuable in itself. Also, since it turns
try {yield expr} finally{}into a syntax error we could revisit the decision in a future edition if somebody actually comes up with compelling use cases.
I'm very uncomfortable with this restriction. Ad hoc restrictions break compositionality. They come back to bite you because you broke generality for want of use cases, when we all know we can't predict all the use cases of a general-purpose programming mechanism. This gets our responsibility backwards: ad hoc restrictions that break generality should be extremely strongly motivated, rather than requiring use cases for the semantics they disallow.
Meanwhile this point:
It's all about
finallywhich up to now has been a strong guarantee.
is wrong. (I've explained this in past meetings, I think at an Apple meeting a year or two ago.) The guarantee of finally does not provide a strong (total) guarantee, it provides a guarantee of the "partial correctness" flavor:
"""
If control reaches the end of the execution of the try block, either by normal completion or abrupt completion, then execute the finally block before proceeding.
"""
A good intuition (and an equivalence that we should strive to preserve) is that yield is acting like a function call to the main continuation, which can choose to return back to the generator or not. This is just the same as if you write:
function infiniteLoop() { while (true); }
try {
infiniteLoop();
} finally {
console.log("and now we are done");
}
We don't guarantee anything about execution of finally blocks if control never reaches the end of their try blocks.
Allen Wirfs-Brock wrote:
In my response to Andy I concluded that syntactically restricting
yieldto not befinallyprotected is the better solution.
It's a shame we have to around the block again. This was discussed over six years ago, when we were prototyping for ES4 and studying Python 2.5. Python started with that restriction and got rid of. So did we for ES4, prototyped in SpiderMonkey and Rhino.
But the rationale based on finally being a strong guarantee is just
broken. No such guarantee, so no need for close.
However (on top of a "But"), dropping close doesn't mean we should ban
yield in try.
I'm also reluctant to ban yield + finally.
yield* should work with any iterable. It is conceptually the same as using
a for-of. If we can afford the call to get the iterator in for-of we can
surely afford it in yield*.
+1 to merge next and send. I don't care about the name.
+1 to getting rid of close.
On May 13, 2013, at 4:22 PM, Brendan Eich wrote:
Allen Wirfs-Brock wrote:
In my response to Andy I concluded that syntactically restricting yield to not be finally protected is the better solution.
It's a shame we have to around the block again. This was discussed over six years ago, when we were prototyping for ES4 and studying Python 2.5. Python started with that restriction and got rid of. So did we for ES4, prototyped in SpiderMonkey and Rhino.
But the rationale based on finally being a strong guarantee is just broken. No such guarantee, so no need for 'close'.
However (on top of a "But"), dropping close doesn't mean we should ban yield in try.
Note I didn't propose no yield's inside of try's, only no yields in try's that include a finally clause.
Allen Wirfs-Brock wrote:
Note I didn't propose no
yields inside oftrys, only noyields intrys that include afinallyclause.
Yes, I was saving typing :-/.
We've been over this at least twice. Let's get it right. No close, yield
in try-with-finally ok.
Merge next and send by letting next take an optional parameter? Ok by me.
Make yield* work on any {next, throw}, not necessary but ok by me too.
Are we there yet?
On May 13, 2013, at 6:11 PM, Brendan Eich <brendan at mozilla.com> wrote:
We've been over this at least twice. Let's get it right. No close, yield in try-with-finally ok.
+1
Merge next and send by letting next take an optional parameter? Ok by me.
+1
Make yield* work on any {next, throw}, not necessary but ok by me too.
Yes with one delta: if there's no .throw it still works, it just defaults to (x) => { throw x }. This way you can write ordinary iterators without having to worry about providing the default throw, and they still function properly as generators.
David Herman wrote:
On May 13, 2013, at 6:11 PM, Brendan Eich<brendan at mozilla.com> wrote:
Merge next and send by letting next take an optional parameter? Ok by me.
+1
I pointed out to Dave that Python has arity checking and did next before adding send in 2.5 for "coroutines", whereas JS has optional params without arity checking, so folding send into next works.
Make yield* work on any {next, throw}, not necessary but ok by me too.
Yes with one delta: if there's no .throw it still works, it just defaults to (x) => { throw x }. This way you can write ordinary iterators without having to worry about providing the default throw, and they still function properly as generators.
+1 or more -- we should not invent new nominal types with stub throw method implementations, people will not use them and they are unnecessary boilerplate.
On May 13, 2013, at 4:15 PM, David Herman <dherman at mozilla.com> wrote:
On May 13, 2013, at 11:07 AM, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
It closes down this whole edge-case focused discussion and that's valuable in itself. Also, since it turns
try {yield expr} finally{}into a syntax error we could revisit the decision in a future edition if somebody actually comes up with compelling use cases.I'm very uncomfortable with this restriction. Ad hoc restrictions break compositionality. They come back to bite you because you broke generality for want of use cases, when we all know we can't predict all the use cases of a general-purpose programming mechanism. This gets our responsibility backwards: ad hoc restrictions that break generality should be extremely strongly motivated, rather than requiring use cases for the semantics they disallow.
And what was I thinking, of course there are use cases: anything where you'd use try/finally in a language with synchronous I/O, you would use try/finally in task.js.
spawn(function*() {
let splashScreen = $("splash");
try {
splashScreen.style.display = 'block';
let assets = yield downloadAssets();
// ...
} catch (e) {
// ...
} finally {
splashScreen.style.display = 'none';
}
});
On May 13, 2013, at 9:44 PM, Brendan Eich wrote:
David Herman wrote:
On May 13, 2013, at 6:11 PM, Brendan Eich<brendan at mozilla.com> wrote:
Merge
nextandsendby letting next take an optional parameter? Ok by me.+1
I pointed out to Dave that Python has arity checking and did
nextbefore addingsendin 2.5 for "coroutines", whereas JS has optional params without arity checking, so foldingsendintonextworks.Make
yield*work on any{next, throw}, not necessary but ok by me too.Yes with one delta: if there's no
.throwit still works, it just defaults to(x) => { throw x }. This way you can write ordinary iterators without having to worry about providing the defaultthrow, and they still function properly as generators.+1 or more -- we should not invent new nominal types with stub
throwmethod implementations, people will not use them and they are unnecessary boilerplate.
All sounds fine with me and I've updated the draft accordingly.
What about providing a convenience resume method on generators to help clarify co-routine style usage? Dave suggested that "resume" was pedagogically useful.
I would define it equivalently two:
resume(...args) {return this.next(...args)}
(resume rather than next delegates to avoid the delegation for normal for-of iterations)
On 14 May 2013 17:07, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
What about providing a convenience
resumemethod on generators to help clarify co-routine style usage? Dave suggested that "resume" was pedagogically useful.I would define it equivalently two:
resume(...args) {return this.next(...args)}(
resumerather thannextdelegates to avoid the delegation for normal for-of iterations)
Not sure about having two names for the same method, but if so, why
not simply make Generator.prototype.resume === Generator.prototype.next?
On May 14, 2013, at 8:12 AM, Andreas Rossberg wrote:
On 14 May 2013 17:07, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:
What about providing a convenience "resume" method on generators to help clarify co-routine style usage? Dave suggested that "resume" was pedagogically useful.
I would define it equivalently two:
resume(...args) {return this.next(...args};
(resume rather than next delegates to avoid the delegation for normal for-of iterations)
Not sure about having two names for the same method, but if so, why not simply make Generator.prototype.resume === Generator.prototype.next?
It could me done that way. I'm not so sure it is such a good practice. GeneratorFunctions are constructors and hence each one provide a unique prototype object for all its instances. So, conceivably a developer can over-ride either or both of "next" and "resume" for a particular family of generators. Specify one as explicitly delegating to the other in the "superclass" makes it clearer which method you need to over-ride if you want to modify both "next" and "resume" behavior. Otherwise, you would always have to remember to over-ride both.
Two names seems like a bad compromise. We should either do next(...args)
or resume(...args). Not both.
Erik Arvidsson wrote:
Two names seems like a bad compromise. We should either do
next(...args)orresume(...args). Not both.
Right, and 'resume' makes no sense for iterators.
C'mon you two whose names start with A: this bikeshedding is wasteful and disharmonious. We have much bigger fish to fry. Now is not the time to be messing around. That is all :-|.
I understand that the consensus among those present at the last TC39 meeting was that iterators should box their return values in objects of the form
{ value: VALUE, done: DONE }where DONE is true or false. Notes here:
rwldrn/tc39-notes/blob/master/es6/2013-03/mar-12.md#412-stopiterationgenerator
For what it's worth (which is very little) this seems like a decent plan to me.
The desugaring of
yield* EXPRwith boxed return values would be:let (g = EXPR) { let received = void 0, send = true; while (true) { let next = send ? g.send(received) : g.throw(received); if (next.done) break; try { received = yield next.value; // *** send = true; } catch (e) { received = e; send = false; } } next.value; }As you can see on the line marked "***", to desugar to plain
yieldyou would have to unbox the value and then allow yield to rebox it. This loses any state onnext-- there could be other properties set onnext, for example ifgis a non-generator.IMHO
yield*should be specified to return the result object as-is, without re-boxing. This precludes a straightforward desugaring, but it is probably more flexible.Hello, I understand that the consensus among those present at the last TC39 meeting was that iterators should box their return values in objects of the form { value: VALUE, done: DONE } where DONE is true or false. Notes here: https://github.com/rwldrn/tc39-notes/blob/master/es6/2013-03/mar-12.md#412-stopiterationgenerator For what it's worth (which is very little) this seems like a decent plan to me. The desugaring of "yield* EXPR" with boxed return values would be: let (g = EXPR) { let received = void 0, send = true; while (true) { let next = send ? g.send(received) : g.throw(received); if (next.done) break; try { received = yield next.value; // *** send = true; } catch (e) { received = e; send = false; } } next.value; } As you can see on the line marked "***", to desugar to plain "yield" you would have to unbox the value and then allow yield to rebox it. This loses any state on "next" -- there could be other properties set on "next", for example if "g" is a non-generator. IMHO yield* should be specified to return the result object as-is, without re-boxing. This precludes a straightforward desugaring, but it is probably more flexible. Andy