for-of loops, IteratorClose and the rest of the iterations in the spec
Why was #1 rejected? I just don't remember.
On 10 September 2014 16:45, Erik Arvidsson <erik.arvidsson at gmail.com> wrote:
I know we talked about this before on several occasions but I'm still unhappy with the current state.
I concur. In particular, it should be possible to implement (the equivalent of) various library functions consuming iterables with for-of. Otherwise, we will likely enter a world with unpredictable discrepancies between ES library functions and user-provided ones.
On 10 September 2014 16:52, Mark S. Miller <erights at google.com> wrote:
Why was #1 rejected? I just don't remember.
I don't remember either, but one counter argument will be that it could be a performance hit. But if we are not willing to take this hit in our "own" functions then we should better not have this feature at all.
This sounds to me like it just need to be reported as a bug There have been previous bugs that have identified places where library methods iterated in a manner that precluded implementing them via for-of. For example ecmascript#2083 Those bugs were all fixed.
Of course it would be nice, if such a bug report actually identified the places where this is an issue. It would be even better if the bug report include the suggest changes to the current algorithms.
does that mean we agreed to the equivalent of Arv's #1? If so, great!
We agree at our June meeting to add the "return" method to generators and to conditionally call "return" (if it is present) when a for-of loop terminates before it exhausts its generator. See rwaldron/tc39-notes/blob/master/es6/2014-06/jun-5.md#conclusionresolution-2 . IteratorClose is part of the implementation of this features and was added in Rev 27 of the spec.
I believe that #1 is implicit in that decision as it has always been an ES6 goal that most library methods should be self-hostable in ES. That means that for-of like iteration in those methods should be specified in a manner that matches the observable semantics of for-of.
It wasn't all that clear to me, but that's good to hear. Does a similar argument apply to constructs like rest and spread?
Yes, except that I don't think the "body" of those loops can actually abnormally exit (or in the case of processing parameter lists, the iterator is an internal list iterator that doesn't have a 'return'). But that code should be reviewed to make sure I don't miss something.
On 9/10/14, 10:45 AM, Erik Arvidsson wrote:
- Add IteratorClose to all places in the spec where we use iterators.
And other specs (e.g. Web IDL) that use iterators, right?
You want aggressive common sub-expression elimination on the specs (all of them, as bz noted).
Is there a common helper spec-only method that can be used both inside ES6 and by other specs?
On Sep 10, 2014, at 10:07 AM, Boris Zbarsky wrote:
And other specs (e.g. Web IDL) that use iterators, right?
Right, or come up with some other way to say: "this loop behaves as if it was implemented using for-of".
On Sep 10, 2014, at 10:24 AM, Brendan Eich wrote:
You want aggressive common sub-expression elimination on the specs (all of them, as bz noted).
Is there a common helper spec-only method that can be used both inside ES6 and by other specs?
Not yet. Probably should be, but I have to prioritize and time is short.
Personally, I would prefer to see W3C specs using ES as their specification language wherever possible. Then for-loops would explicitly be for-of loops.
On 9/10/14, 1:28 PM, Allen Wirfs-Brock wrote:
Right, or come up with some other way to say: "this loop behaves as if it was implemented using for-of".
Thing is, for my Web IDL use case I don't want quite the behavior of for-of. I want something like this:
- Let method be the result of CheckIterable(V).
- ReturnIfAbrupt(method).
- If IsCallable(method) is false, go off and do something else, since V is not iterable.
- Let iter be GetIterator(V, method).
- ReturnIfAbrupt(iter).
and then IteratorStep my way through "iter".
An additional complication here is that I'm not using this in a situation where the return value of my algorithm is an ES6 completion (so the ReturnIfAbrupt bits above there are actually somewhat bunk; they should be something like "if method is an abrupt completion, then propagate method.[[value]] as an exception" or some such; we need to figure out how to make the way DOM/IDL talk about exceptions really interface well with the way ES does). So I'm not quite sure what IteratorClose should actually do in my setting, assuming it should be called at all.
On Sep 10, 2014, at 10:37 AM, Boris Zbarsky wrote:
On 9/10/14, 1:28 PM, Allen Wirfs-Brock wrote:
Right, or come up with some other way to say: "this loop behaves as if it was implemented using for-of".
Thing is, for my Web IDL use case I don't want quite the behavior of for-of. I want something like this:
- Let method be the result of CheckIterable(V).
- ReturnIfAbrupt(method).
- If IsCallable(method) is false, go off and do something else, since V is not iterable.
- Let iter be GetIterator(V, method).
- ReturnIfAbrupt(iter).
and then IteratorStep my way through "iter".
So, try expressing this in JS. That's what a JS help-hosting compliant implementation is going to have to do.
An additional complication here is that I'm not using this in a situation where the return value of my algorithm is an ES6 completion (so the ReturnIfAbrupt bits above there are actually somewhat bunk; they should be something like "if method is an abrupt completion, then propagate method.[[value]] as an exception" or some such; we need to figure out how to make the way DOM/IDL talk about exceptions really interface well with the way ES does). So I'm not quite sure what IteratorClose should actually do in my setting, assuming it should be called at all.
IteratorClose, given an abrupt completion and an iterator, calls the iterator's 'return' method if such a method exists. It then proceeds with the abrupt completion (subject to some additional exception juggling in case the 'return' method also throws. ). It all in the IteratorClose spec.
On 9/10/14, 1:47 PM, Allen Wirfs-Brock wrote:
So, try expressing this in JS. That's what a JS help-hosting compliant implementation is going to have to do.
Sure.
var method = V[Symbol.iterator];
if (typeof method != "function") {
// Not iterable; do something else
}
var iter = method.call(iter);
and then go through calling .next() and so on.
Now in this self-hosted case, how do I express IteratorClose? Seems to me like it's just a matter of wrapping the relevant bits (which ones, though?) in a try/catch and doing this in the catch clause:
catch (e) {
if ("return" in iter) {
iter.return();
}
throw e;
}
That ought to match the semantics of IteratorClose, yes?
On Sep 10, 2014, at 10:58 AM, Boris Zbarsky wrote:
Now in this self-hosted case, how do I express IteratorClose?
why wouldn't a self-hoster just continue with:
for (let next of iter} {
/* body */
}
which will do all the IteratorClose work for them. In fact, isn't that how we want a self-hoster to code it. Anything either ES or WebIDL does that is hostile to that pattern is counter productive as self-hosters are probably going to do that anyway, even if the semantics is a slight mismatch to the spec.
Seems to me like it's just a matter of wrapping the relevant bits (which ones, though?
whatever constitutes the loop "body"
) in a try/catch and doing this in the catch clause:
catch (e) { if ("return" in iter) { iter.return(); } throw e; }That ought to match the semantics of IteratorClose, yes?
You also will need a finally clause, if the loop body contains any explicit returns.
On 9/10/14, 3:20 PM, Allen Wirfs-Brock wrote:
why wouldn't a self-hoster just continue with:
for (let next of iter} { /* body */ }
I feel like I'm missing something here.
"iter" is an iterator, not an iterable. So I don't think this will do the right thing, unless I'm seriously missing something.
Anything either ES or WebIDL does that is hostile to that pattern is counter productive as self-hosters are probably going to do that anyway, even if the semantics is a slight mismatch to the spec.
What I am trying to define in Web IDL is the semantics of methods that take an argument which is either an iterable or something else. That means they need to check whether the given thing is iterable, and if so iterate, otherwise do something else. See the thread that ended up in lists.w3.org/Archives/Public/public-script-coord/2013OctDec/0370.html and the algorithm in that mail; that's what I'm specifying.
whatever constitutes the loop "body"
OK. That's fairly straightforward.
You also will need a finally clause, if the loop body contains any explicit returns.
It only does when the iterator claims there are no more things to be had, in which case do I still want to be calling the return() thing?
On Sep 10, 2014, at 12:41 PM, Boris Zbarsky wrote:
I feel like I'm missing something here.
"iter" is an iterator, not an iterable. So I don't think this will do the right thing, unless I'm seriously missing something.
In practice (because of for-of semantics) Iterators need to be Iterables that return themselves as the associated iterator. All built-in iterators in ES have this characteristic
What I am trying to define in Web IDL is the semantics of methods that take an argument which is either an iterable or something else. That means they need to check whether the given thing is iterable, and if so iterate, otherwise do something else. See the thread that ended up in lists.w3.org/Archives/Public/public-script-coord/2013OctDec/0370.html and the algorithm in that mail; that's what I'm specifying.
For example, Array.from does that kind of argument discrimination: people.mozilla.org/~jorendorff/es6-draft.html#sec-array.from
It only does when the iterator claims there are no more things to be had, in which case do I still want to be calling the return() thing?
No, you only call 'return' when exiting the loop before the iterator has indicated that it is 'done'..
On 9/10/14, 4:11 PM, Allen Wirfs-Brock wrote:
In practice (because of for-of semantics) Iterators need to be Iterables that return themselves as the associated iterator. All built-in iterators in ES have this characteristic
Ah, I see. But that's observably different from just doing the loop manually, since someone can change things like %ArrayIteratorPrototype% [@@iterator], right?
For example, Array.from does that kind of argument discrimination: people.mozilla.org/~jorendorff/es6-draft.html#sec-array.from
Yes, I'm aware. I was looking at that earlier today while writing up heycam/webidl#18.
It slightly differs from what the outcome of the Web IDL discussion was in that it commits to an iterator if CheckIterable returns anything other than undefined, whereas the outcome of the thread I linked earlier was that we want to commit if CheckIterable returns a callable... It would make some sense to align the two; at this point I can't recall why it was we decided to do the callable check in the IDL version, nor do I know why the ES spec decided to not do a callable check in this situation.
Other than that, I basically copied what Array.from/TypedArrayFrom does, yes. Including the not calling IteratorClose bit. ;)
Just for the sake of completeness, here is complete translation of
for-of to ES5 syntax.
ES6 for-of statement:
for (var value of iterable) {
if (doReturn) return;
if (doBreak) break;
if (doContinue) continue;
otherExpression;
}
Translated to ES5 syntax:
var $iterable = iterable;
if ($iterable === undefined || $iterable === null) {
throw new TypeError();
}
var $iterator = Object($iterable)[Symbol.iterator]();
if (Object($iterator) !== $iterator) {
throw new TypeError();
}
while (true) {
var $nextResult = $iterator.next();
if (Object($nextResult) !== $nextResult) {
throw new TypeError();
}
if ($nextResult.done) {
break;
}
var $nextValue = $nextResult.value;
var $callClose = false;
try {
value = $nextValue;
if (doReturn) { $callClose = true; return; }
if (doBreak) { $callClose = true; break; }
if (doContinue) continue;
otherExpression;
} catch ($exception) {
try {
if ("return" in $iterator) {
$iterator.return();
}
} catch ($ignore) {
// ignore
}
throw $exception;
} finally {
if ($callClose) {
if ("return" in $iterator) {
$iterator.return();
}
}
}
}
From: es-discuss [mailto:es-discuss-bounces at mozilla.org] On Behalf Of Boris Zbarsky
On 9/10/14, 10:45 AM, Erik Arvidsson wrote:
- Add IteratorClose to all places in the spec where we use iterators.
And other specs (e.g. Web IDL) that use iterators, right?
Relatedly, it would be really nice to have a spec-level for-of, both for ES and for web specs.
(Not to mention a spec-level try/catch, but that's another story...)
On 9/10/14, 4:57 PM, Domenic Denicola wrote:
Relatedly, it would be really nice to have a spec-level for-of, both for ES and for web specs.
You mean so people don't have to write things like heycam.github.io/webidl/#add-map-elements-from-iterable by hand? Yeah, it would.
(Not to mention a spec-level try/catch, but that's another story...)
Abrupt completions sort of give you that in ES. Or did you mean so you don't have to sprinkle ReturnIfAbrupt() everywhere?
From: Boris Zbarsky [mailto:bzbarsky at mit.edu]
On 9/10/14, 4:57 PM, Domenic Denicola wrote:
Relatedly, it would be really nice to have a spec-level for-of, both for ES and for web specs.
You mean so people don't have to write things like heycam.github.io/webidl/#add-map-elements-from-iterable by hand?
Sorry, my mail client didn't see the threading on this, so after getting through the rest of my emails I realize that's what much of the rest of the conversation was about. But yes, exactly :)
For heycam/webidl#18, I think it would be ideal to use exact for-of if possible, perhaps after an initial discrimination, instead of manually doing all those things to the iterable.
So yeah, ideally we should define something so that specs can say
- For <var>x</var> of <var>someIterable</var>,
- Use <var>x</var> in some way.
Perhaps WebIDL is the best place to do that for now.
(Not to mention a spec-level try/catch, but that's another story...)
Abrupt completions sort of give you that in ES. Or did you mean so you don't have to sprinkle ReturnIfAbrupt() everywhere?
Right. ReturnIfAbrupt always felt to me like an overreaction to the problem of underspecified exception handling behavior. Something like: "Try: (... substeps ...) Catch e: ( ... substeps ... )" would be quite nice, at times.
Relatedly, I am documenting a variety of small deviations from ES's algorithm conventions that I plan to use while writing the stream spec. We should probably incorporate these into either ES or WebIDL at some point. The intention is to lose some of the verbosity of the ES conventions to the extent that they hurt clarity, while staying within the realm of precision so that the algorithms are unambiguously interpreted by all implementers. Feedback welcome, probably on the repo's issue tracker. Even if the feedback is "this is a bad idea; just do things exactly like ES does to minimize dialects."
On 9/10/14, 5:12 PM, Domenic Denicola wrote:
For heycam/webidl#18, I think it would be ideal to use exact for-of if possible, perhaps after an initial discrimination, instead of manually doing all those things to the iterable.
Again, I basically copied Array.from, except for the IsCallable check...
Perhaps WebIDL is the best place to do that for now.
For what it's worth, things that consume Web IDL and just want to snapshot the iterable will be able to do exactly that with sequence types.
If they want to interleave iteration and operation, of course, they'll need something else.
Right. ReturnIfAbrupt always felt to me like an overreaction to the problem of underspecified exception handling behavior. Something like: "Try: (... substeps ...) Catch e: ( ... substeps ... )" would be quite nice, at times.
I'd love for us to get there, while aligning the various web specs (ES on the one hand and Web IDL and the various W3C/WHATWG specs on the other)...
On Sep 10, 2014, at 2:12 PM, Domenic Denicola wrote:
From: Boris Zbarsky [mailto:bzbarsky at mit.edu]
On 9/10/14, 4:57 PM, Domenic Denicola wrote:
Relatedly, it would be really nice to have a spec-level for-of, both for ES and for web specs.
You mean so people don't have to write things like heycam.github.io/webidl/#add-map-elements-from-iterable by hand?
Sorry, my mail client didn't see the threading on this, so after getting through the rest of my emails I realize that's what much of the rest of the conversation was about. But yes, exactly :)
For heycam/webidl#18, I think it would be ideal to use exact for-of if possible, perhaps after an initial discrimination, instead of manually doing all those things to the iterable.
So yeah, ideally we should define something so that specs can say
- For <var>x</var> of <var>someIterable</var>,
- Use <var>x</var> in some way.
Perhaps WebIDL is the best place to do that for now.
(Not to mention a spec-level try/catch, but that's another story...)
Abrupt completions sort of give you that in ES. Or did you mean so you don't have to sprinkle ReturnIfAbrupt() everywhere?
Right. ReturnIfAbrupt always felt to me like an overreaction to the problem of underspecified exception handling behavior. Something like: "Try: (... substeps ...) Catch e: ( ... substeps ... )" would be quite nice, at times.
So you want a language with structured control constructs. We actually have such a language - its called ECMAScript :-)
But, really the ES spec (at least chapters 6-15) has to operate using primitive abstractions like abrupt completions, etc. because it is defining the execution model of the higher level structured controls constructs (including closures, etc.)
But I don't see any reason why most of what is defined in WebIDL has to be specified using those low level control abstractions. Why not just us ES as the specification language for most things in the WebIDL world? Or if not full ES, a simplified form "Spec-ES".
In the long run, we could probably also do this for many of the ES built-ins but there is no pressing need do such a rewrite at this time. But I don't see why new WebIDL level specs. would want to burden themselves with such a low level notation. And I really hope that most APIs spec'ed using WebIDL are designed to be implementable using ES.
On 9/10/14, 5:48 PM, Allen Wirfs-Brock wrote:
Why not just us ES as the specification language for most things in the WebIDL world? Or if not full ES, a simplified form "Spec-ES".
There are a few issues with using ES for specifying Web IDL. It's possible that these can be overcome, of course.
First, ES doesn't actually have some of the data types involved. There is no record type that would correspond to an IDL dictionary, for example. You could use objects, or Maps to represent those, but would need to make it very clear that those are not using the page-modifiable prototypes, while continuing to operate in the page Realm in some sense.
Second, and related to the first item, Web IDL needs to operate on property descriptors, because it needs to define the behavior of some internal operations like [[Get]] and [[GetOwnProperty]]. This comes back to not having records; operations on descriptor objects are not the same thing, again unless you assume clean prototypes somehow.
Third, a number of the desired operations (ToUint32, ToInt32, ToNumber, ToBoolean, ToString, ToObject, Type) are somewhere betwee "a pain" and "starts to look like an obfuscated programming contest" when you try to do them in ES proper. So whatever specification formalism we use needs to be able to reference these directly. A Spec-ES could do that, of course.
Fourth, any algorithm that uses lists would need to make it clear that the lists use the magic clean Array.prototype while arrays in the same algorithm might use the normal page-modifiable prototype.
Fifth, Web IDL needs to be able to call [[DefineOwnProperty]] and the like. This can be done via Object.defineProperty as long as you once again make it very clear which operations are using a clean Object and which are not.
So I guess my main issue here is the same one that self-hosted builtin implementations have in V8 and SpiderMonkey: you have to be really careful to distinguish between your usage of clean built-in stuff and page-visible stuff. The result can be fairly taxing to write, read, reason about, and modify. It's doable, but I'm not sure whether the end result would be better than what we have now. :(
In the long run, we could probably also do this for many of the ES built-ins
This would be a good exercise for dealing with some of the above sorts of problems, yes....
And I really hope that most APIs spec'ed using WebIDL are designed to be implementable using ES.
Most APIs I've seen people adding recently really aren't, because they seem to be about low-level hardware access and whatnot. For what it's worth.
On Sep 10, 2014, at 6:24 PM, Boris Zbarsky wrote:
On 9/10/14, 5:48 PM, Allen Wirfs-Brock wrote:
Why not just us ES as the specification language for most things in the WebIDL world? Or if not full ES, a simplified form "Spec-ES".
There are a few issues with using ES for specifying Web IDL. It's possible that these can be overcome, of course.
To clarify, I wasn't necessarily talking about using ES to specify Web IDL. Rather I meant using ES as an alternative to pseudo code for specifying the semantics of APIs defined with Web IDL interfaces.
To support that I assume that you would have to defined some WebIDL specific functions that would be callable from those ES-based specifications.
First, ES doesn't actually have some of the data types involved. There is no record type that would correspond to an IDL dictionary, for example. You could use objects, or Maps to represent those, but would need to make it very clear that those are not using the page-modifiable prototypes, while continuing to operate in the page Realm in some sense.
Yes, you can abstract any sort of data structure you need as spec. level functions and objects and restrict them in whatever way you need to.
That was the sort of thing I was thinking about when I allude to a "Spec-ES".
Second, and related to the first item, Web IDL needs to operate on property descriptors, because it needs to define the behavior of some internal operations like [[Get]] and [[GetOwnProperty]]. This comes back to not having records; operations on descriptor objects are not the same thing, again unless you assume clean prototypes somehow.
Third, a number of the desired operations (ToUint32, ToInt32, ToNumber, ToBoolean, ToString, ToObject, Type) are somewhere betwee "a pain" and "starts to look like an obfuscated programming contest" when you try to do them in ES proper. So whatever specification formalism we use needs to be able to reference these directly. A Spec-ES could do that, of course.
Fourth, any algorithm that uses lists would need to make it clear that the lists use the magic clean Array.prototype while arrays in the same algorithm might use the normal page-modifiable prototype.
Fifth, Web IDL needs to be able to call [[DefineOwnProperty]] and the like. This can be done via Object.defineProperty as long as you once again make it very clear which operations are using a clean Object and which are not.
So I guess my main issue here is the same one that self-hosted builtin implementations have in V8 and SpiderMonkey: you have to be really careful to distinguish between your usage of clean built-in stuff and page-visible stuff. The result can be fairly taxing to write, read, reason about, and modify. It's doable, but I'm not sure whether the end result would be better than what we have now. :(
I think the main benefit is that you get closures, exception handling, for-of, etc. Getting those things correct (and ES implementable) at the pseudo-code level is challenging.
In the long run, we could probably also do this for many of the ES built-ins
This would be a good exercise for dealing with some of the above sorts of problems, yes....
And I really hope that most APIs spec'ed using WebIDL are designed to be implementable using ES.
Most APIs I've seen people adding recently really aren't, because they seem to be about low-level hardware access and whatnot. For what it's worth.
Sure hardware accesses need to be abstracted, isn't there typically a layer of post processing or data packaging wrapped around the direct hardware access. Higher level access to iterators, for-of , closures, exception would be very useful in those layers.
On 9/11/14, 1:47 PM, Allen Wirfs-Brock wrote:
To clarify, I wasn't necessarily talking about using ES to specify Web IDL. Rather I meant using ES as an alternative to pseudo code for specifying the semantics of APIs defined with Web IDL interfaces.
Ah, I see.
Yeah, I'm not sure how much pseudo code actually happens when defining those APIs. Typically what you get are some sort of lists of prose steps.
isn't there typically a layer of post processing or data packaging wrapped around the direct hardware access.
Pretty thin, but somewhat, yes.
For or loops are spec'ed to call the internal spec function, IteratorClose when there is an abrupt completion in the loop body (an exception was thrown, return and break)
The point of this was to allow cleaning up the iterator in case it holds on to some kind of resource.
The problem is that none of the other iterations in the spec calls IteratorClose. For example,
class MySet extends Set { add(v) { if (valueNoGood(v)) throw new NoGoodError(); return super(v); } } new MySet(new ReadLines('~/test.txt'));In the above code
ReadLines.prototype.returnis never called, defeating the whole purpose of adding return hooks to for-of.I see two options here.
I know we talked about this before on several occasions but I'm still unhappy with the current state.
For or loops are spec'ed to call the internal spec function, IteratorClose when there is an abrupt completion in the loop body (an exception was thrown, return and break) The point of this was to allow cleaning up the iterator in case it holds on to some kind of resource. The problem is that none of the other iterations in the spec calls IteratorClose. For example, ```js class MySet extends Set { add(v) { if (valueNoGood(v)) throw new NoGoodError(); return super(v); } } new MySet(new ReadLines('~/test.txt')); ``` In the above code `ReadLines.prototype.return` is never called, defeating the whole purpose of adding return hooks to for-of. I see two options here. 1. Add IteratorClose to all places in the spec where we use iterators. 2. Remove IteratorClose from for-of loops. I know we talked about this before on several occasions but I'm still unhappy with the current state. -- erik -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.mozilla.org/pipermail/es-discuss/attachments/20140910/6772451a/attachment.html>