Nov 18 notes
On Nov 18, 2010, at 4:08 PM, Waldemar Horwat wrote:
Consensus that we should have iterators.
For this, after all these years (JS1.7 added meta-programmable for-in in 2006), I'm grateful, although I wanted to add something your notes did not report:
To get consensus, we made a tentative agreement to leave for-in as it was and not enable any meta-programmability of it, consigning it to the historically-underspecified and non-interoperable enumeration mystery-meat status.
Instead, we did the committee-plays-it-safe thing of inventing new syntax for meta-programmable iteration:
for (var i : x) ...
This is a break from Python and JS1.7+ in SpiderMonkey and Rhino -- although it matches very late Java and C++ extensions that are similar (but not the same), and really not relevant to JS.
Worse, the use of colon in this new for syntax is confusingly similar to long-standing future-proofing intentions around runtime type annotations (aka guards or contracts).
(BTW, I don't think :: is a good type annotation or guard punctuator, btw -- it's the C++ namespace qualification operator, also specified for namespacing by ECMA-357 (E4X) and used that way in ActionScript 3 (and in ES4, RIP). So I was surprised to see :: used for annotation-like syntax in strawman:guards and strawman:trademarks.)
for (var i : x) ... // must be new iteration for (var i : T : x) ... // iteration again, but parsed how? for (var i : T in x) ... // for-in with annotated var
Depending on what T might be, grammatically, this could get sticky for top-down parsers. It is confusing and ugly in any event, IMHO.
Probably we need to take our time and not rush into a meta-programming-here syntax variant of for-in. I'll not propose anything better right now.
On Nov 21, 2010, at 7:21 PM, Chris Sells wrote:
I'm a huge fan of iterators. As far as index goes, I'm not a fan of the use of the colon. We should take our time, but something that I wanted to throw into the pot would be to build on the key-based nature of for-in with syntax like this:
forvals ( var x in xs ) {...}
Also, have we given any thought to making it easier to implement custom iterators ala the C# "yield return" operator?
We've prototyped and shipped generators based on Python 2.5 (with a simplification that avoids introducing GeneratorExit) for years, in SpiderMonkey and Rhino. The yield operator is a low-precedence expression prefix whose presence in a function makes it a generator. Wiki link:
doku.php?id=strawman:generators&s=generators
This strawman wil be revised based
On Nov 21, 2010, at 7:21 PM, Chris Sells wrote:
I'm a huge fan of iterators. As far as index goes, I'm not a fan of the use of the colon. We should take our time, but something that I wanted to throw into the pot would be to build on the key-based nature of for-in with syntax like this:
forvals ( var x in xs ) {...}
Resist bikeshedding early and all that, but there's already a strike against this or anything like it:
JS lacks run-together keywords apart from typeof and instanceof, which do not abbreviate as "vals" does.
In grasping for ":" instead of "in", the committee was trying to keep the introductory keyword "for". I think we had agreement that sticking with "for" is still worth using as a syntax design guide or "back up and try another route forward [than 'in']" design point.
On Nov 21, 2010, at 7:41 PM, Brendan Eich wrote:
On Nov 21, 2010, at 7:21 PM, Chris Sells wrote:
I'm a huge fan of iterators. As far as index goes, I'm not a fan of the use of the colon. We should take our time, but something that I wanted to throw into the pot would be to build on the key-based nature of for-in with syntax like this:
forvals ( var x in xs ) {...}
Resist bikeshedding early and all that, but there's already a strike against this or anything like it:
JS lacks run-together keywords apart from typeof and instanceof, which do not abbreviate as "vals" does.
In grasping for ":" instead of "in", the committee was trying to keep the introductory keyword "for". I think we had agreement that sticking with "for" is still worth using as a syntax design guide or "back up and try another route forward [than 'in']" design point.
And (sorry, I'll try to keep replies in one message next time) "for" vs. "forvals" does not exactly scream "keys" vs. "values", since "for" is only about "keys" if you know ECMA-262 and expect the mystery meat of enumeration.
If we manage to make normative specifications for enumeration, which has been a goal recently, we'll still have "for-in" for that "keys" meaning. So there won' t be any "keys" : "vals" symmetry to play off of, just some longer or different syntax for meta-programmable iteration.
Which raises another point: meta-programmable iteration is not necessarily about "values" and not "keys". A custom iterator could (and in the strawman does) return key/value pairs. The whole keys vs. values dilemma is a false one here.
And (sorry, I'll try to keep replies in one message next time) "for" vs. "forvals" does not exactly scream "keys" vs. "values", since "for" is only about "keys" if you know ECMA-262 and expect the mystery meat of enumeration.
IMO, "forvals" is a non-starter, as is "foreach" or "for each". The "for" part of the syntax denotes quantification, and the stuff to the right of the variable denotes what is being iterated. In all cases, we are talking about universal quantification, so they should all be "for".
Which raises another point: meta-programmable iteration is not necessarily about "values" and not "keys". A custom iterator could (and in the strawman does) return key/value pairs. The whole keys vs. values dilemma is a false one here.
Indeed. There are an unbounded number of types of sequences that can be iterated over. Whether we provide 2 or 200 it will never be enough. Hence the need for a general, programmable iteration mechanism.
for (var i : x) ... // must be new iteration for (var i : T : x) ... // iteration again, but parsed how? for (var i : T in x) ... // for-in with annotated var
Bummer!
I'm beginning to feel more strongly again that overloading for-in, as opposed to introducing yet another syntactic variant of `for', is the right way to go.
<thought experiment>
Imagine we made Harmony backwards-incompatible in the following way: for-in loops would only work on iterable objects, and would dynamically fail on non-iterable objects. So if you wanted the legacy keys behavior, you would have to explicitly call a `keys' library function. </thought experiment>
Now say we make for-in meta-programmable in the backwards-compatible way: non-iterable objects default to the legacy keys' behavior, and iterable objects use theiterate' trap. Then old code will migrate and just work out of the box most of the time, and lazy programmers will get programs that just work out of the box most of the time. The only hazard is that you occasionally meant to get keys and instead get the `iterate' behavior. How often do we really expect this will a problem? And how often do we really expect it'll pass through undetected? Even in the cases where you mistakenly write |for (x in obj)| and get the iteration behavior, in many (most?) cases x will get assigned values of some unexpected type and the program will fail pretty quickly.
I have never claimed that any solution is perfect. But I still think that the hazards involved in overloading the for-in syntax are less costly than the proliferation of additional syntactic forms.
Let me make yet another argument in favor of the overloaded for-in: if we had a new form that works better than for-in, I would tell people that they should just deprecate for-in and replace it with
for (x <<whatevermagicsigil>> keys(obj))
Why? It's more readable and explicit, and they only ever have to learn one form in the language. By contrast, if we allowed for-in to be overloaded, I would tell people that they should deprecate the legacy for-in and replace it with an explicit iterator such as:
for (x in keys(obj))
And yet, there wouldn't be a proliferation of language forms, and we'd get to use the excellent "in" syntax. Deprecating a truly sweet syntax would suck.
On Mon, Nov 22, 2010 at 8:37 AM, David Herman <dherman at mozilla.com> wrote:
for (var i : x) ... // must be new iteration for (var i : T : x) ... // iteration again, but parsed how? for (var i : T in x) ... // for-in with annotated var
Bummer!
I'm beginning to feel more strongly again that overloading for-in, as opposed to introducing yet another syntactic variant of `for', is the right way to go.
<thought experiment> Imagine we made Harmony backwards-incompatible in the following way: for-in loops would only work on iterable objects, and would dynamically fail on non-iterable objects. So if you wanted the legacy keys behavior, you would have to explicitly call a `keys' library function. </thought experiment>
If we're gonna go invent new keywords why not use the obvious?
iterate (var x in y) ...
On Nov 22, 2010, at 12:49 AM, Peter van der Zee wrote:
On Mon, Nov 22, 2010 at 8:37 AM, David Herman <dherman at mozilla.com> wrote:
for (var i : x) ... // must be new iteration for (var i : T : x) ... // iteration again, but parsed how? for (var i : T in x) ... // for-in with annotated var
Bummer!
I'm beginning to feel more strongly again that overloading for-in, as opposed to introducing yet another syntactic variant of `for', is the right way to go.
<thought experiment> Imagine we made Harmony backwards-incompatible in the following way: for-in loops would only work on iterable objects, and would dynamically fail on non-iterable objects. So if you wanted the legacy keys behavior, you would have to explicitly call a `keys' library function. </thought experiment>
If we're gonna go invent new keywords why not use the obvious?
iterate (var x in y) ...
Did you read Dave's previous post, or mine? "We" are not going to go invent some new overlong non-reserved word. "for" is the right word.
One might try retasking "do" but it's ambiguous: do-while loops do not require braces (K&R style favors them even for one-liners but it's just a convention).
Dave's point about getting people to use one universal quantifying loop structure is good. If that loop syntax is not for-in, it will have to be something new that works even when there is no iterate trapping proxy. But that will just be a suckier version of the universal (enumerating if not custom-iterating) for-in from JS1.7.
So what good are we really doing by forking for-in into bastard children (the one we have, definitely a bastard -- I should know! :-/) and a new one with red hair?
My arguments in favor of keeping the existing "for-in" syntax and making it meta-programmable:
- Simplicity. Don't underestimate the complexity creep of introducing a new looping construct. Many small changes add up quickly. Especially for novices, having two looping constructs that are so similar will be terribly confusing.
- Keeping the existing syntax means that the tons of JS code already out there can instantly become client-code to iterators. As Brendan noted: you can use iterators to generate a stream of keys (strings) without confusing clients.
- Client code to proxies that emulate objects with lots of properties can continue to use for-in. It just feels wrong that client code should change because of pure implementation details of an object (i.e. whether it eagerly or lazily generates its keys).
Of course, if an object changes its behavior from iterating keys to iterating values, this breaks clients and they should be modified to use "for (var k in keys(obj))". But I don't see how this differs from any other such changes made to an object. The important thing to note here is that turning an object into an iterator requires explicit action by the programmer. If iterators were implemented ala Python using a magical "iterate" hook, then I'd complain because Harmony code could silently turn normal objects into iterators. But there's no such risk with this proposal.
I think that's a key point worth re-iterating: iterators and regular objects are sufficiently distinct so that there's no risk of automatically converting one into the other. There is only a risk to existing client-code if a Harmony programmer changes a normal object into an iterator. But at that point the programmer knows he's making a non-upwards-compatible change and clients should be changed accordingly. I don't see how the for-in case differs in this respect, fundamentally, from say, renaming a method.
Cheers, Tom
2010/11/22 Brendan Eich <brendan at mozilla.com>
On Nov 22, 2010, at 2:08 AM, Tom Van Cutsem wrote:
My arguments in favor of keeping the existing "for-in" syntax and making it meta-programmable:
- Simplicity. Don't underestimate the complexity creep of introducing a new looping construct. Many small changes add up quickly. Especially for novices, having two looping constructs that are so similar will be terribly confusing.
You're not saving the addition of a looping construct, in making for-in behave differently depending on what is on the right hand side of 'in' you're merely adding an additional looping that is not syntactically observable.
- Keeping the existing syntax means that the tons of JS code already out there can instantly become client-code to iterators. As Brendan noted: you can use iterators to generate a stream of keys (strings) without confusing clients.
And all existing standards compliant code can no longer rely on for in doing what it has done for years. Suddently for in behaviour may change based on the prototype chain.
- Client code to proxies that emulate objects with lots of properties can continue to use for-in. It just feels wrong that client code should change because of pure implementation details of an object (i.e. whether it eagerly or lazily generates its keys).
My understanding was that proxies would have a trap to allow them to generate an array of strings to be used.
Of course, if an object changes its behavior from iterating keys to iterating values, this breaks clients and they should be modified to use "for (var k in keys(obj))". But I don't see how this differs from any other such changes made to an object. The important thing to note here is that turning an object into an iterator requires explicit action by the programmer. If iterators were implemented ala Python using a magical "iterate" hook, then I'd complain because Harmony code could silently turn normal objects into iterators. But there's no such risk with this proposal.
I think that's a key point worth re-iterating: iterators and regular objects are sufficiently distinct so that there's no risk of automatically converting one into the other. There is only a risk to existing client-code if a Harmony programmer changes a normal object into an iterator. But at that point the programmer knows he's making a non-upwards-compatible change and clients should be changed accordingly. I don't see how the for-in case differs in this respect, fundamentally, from say, renaming a method.
I do not expect the behaviour of for(in) to change from harmony to non-harmony code. Ignoring all other concerns it would mean the behaviour of objects passed from a harmony context to a non-harmony context would be unexpected.
I'm kind of frustrated that after discussing all this last week, and apparently getting to some kind of consensus, we've more or less instantly gone back to what we had at the beginning of the meeting, where for(in) gets magically overloaded and a developer has some magical way to understand what a given for(in) loop is going to do. If this was going to be the outcome why did we even spend time discussing this?
On Nov 22, 2010, at 11:14 AM, Oliver Hunt wrote:
On Nov 22, 2010, at 2:08 AM, Tom Van Cutsem wrote:
My arguments in favor of keeping the existing "for-in" syntax and making it meta-programmable:
- Simplicity. Don't underestimate the complexity creep of introducing a new looping construct. Many small changes add up quickly. Especially for novices, having two looping constructs that are so similar will be terribly confusing.
You're not saving the addition of a looping construct, in making for-in behave differently depending on what is on the right hand side of 'in' you're merely adding an additional looping that is not syntactically observable.
Is the additional aspect important enough to split syntax over?
If so, would you make the new form only work on proxies and throw given a non-proxy on the right of "in"?
If not, why not?
- Keeping the existing syntax means that the tons of JS code already out there can instantly become client-code to iterators. As Brendan noted: you can use iterators to generate a stream of keys (strings) without confusing clients.
And all existing standards compliant code can no longer rely on for in doing what it has done for years. Suddently for in behaviour may change based on the prototype chain.
That's true in Harmony, and as you noted on the list a while ago, also true in ES5 strict. I decried too much migration tax but allowed we want to break compatibility for important wins. Lexical scope all the way up is one such proposed win, justifying removing the global object from the scope chain.
In my view, letting for-in be reformed by library authors is another case where the migration tax is worth it.
Now, I need to ask whether you are making an absolute statement: Harmony must be runtime as well as syntactically compatible with ES5 strict, i.e., a superset language?
- Client code to proxies that emulate objects with lots of properties can continue to use for-in. It just feels wrong that client code should change because of pure implementation details of an object (i.e. whether it eagerly or lazily generates its keys).
My understanding was that proxies would have a trap to allow them to generate an array of strings to be used.
That's true, the fundamental trap is enumerate, but as noted on the wiki and pointed out by Waldemar when we reviewed proxies and moved them to harmony:proposals status, this does not work well for "large" and "infinite" objects.
We moved proxies to harmony:proposals status with the agreement that the iteration protocol would address such hard cases. The iterators proposal does that, including details about proxies as prototypes of other objects (you don't want to switch to iterate if you start with enumeration -- you must call the proxy handler's enumerate trap).
Last week we agreed toward the end of the meeting, with some lack of clarity about how to do this, to recast for (x in y) as for (x : keys(y)) and allow either enumerate or (if provided) iterate to be used to handle large/infinite objects.
Now the for-: syntax looks like a mistake, and we still haven't reworked things to address the required large/infinite cases.
I'm reviewing all this because I do not think everyone has kept up with the details. But the details matter, and they have some irreducible complexity we can't wish away. They motivate more than just the enumerate trap which eagerly returns all the property keys
Of course, if an object changes its behavior from iterating keys to iterating values, this breaks clients and they should be modified to use "for (var k in keys(obj))". But I don't see how this differs from any other such changes made to an object. The important thing to note here is that turning an object into an iterator requires explicit action by the programmer. If iterators were implemented ala Python using a magical "iterate" hook, then I'd complain because Harmony code could silently turn normal objects into iterators. But there's no such risk with this proposal.
I think that's a key point worth re-iterating: iterators and regular objects are sufficiently distinct so that there's no risk of automatically converting one into the other. There is only a risk to existing client-code if a Harmony programmer changes a normal object into an iterator. But at that point the programmer knows he's making a non-upwards-compatible change and clients should be changed accordingly. I don't see how the for-in case differs in this respect, fundamentally, from say, renaming a method.
I do not expect the behaviour of for(in) to change from harmony to non-harmony code. Ignoring all other concerns it would mean the behaviour of objects passed from a harmony context to a non-harmony context would be unexpected.
That is a risk but it is not an absolute. It is one end of a trade-off. The other end is the benefit of avoiding new and hard-to-make-winning syntax, splitting for-in and for-: by dynamic type of what is on the right-hand side of in vs. :, complicating the surface language, and precluding library authors from helping in the near term.
I'm kind of frustrated that after discussing all this last week, and apparently getting to some kind of consensus, we've more or less instantly gone back to what we had at the beginning of the meeting, where for(in) gets magically overloaded and a developer has some magical way to understand what a given for(in) loop is going to do.
"We" haven't gone back. The agreement at the meeting was fragile, a rushed compromise to overcome an objection that not everyone believes is an absolute that trumps the benefit in the trade-off. So we are revisiting. Recall too that Tom was not on the call for this part of the meeting, due to his time zone.
Frustration over this opening up again is not going to do much to alter the fundamentals. We need to keep arguing until we have shared premises and shared weights for costs and benefits. If possible.
If this was going to be the outcome why did we even spend time discussing this?
Because we were trying to reach agreement. Who says failure is not an option? Failure is important. It also is not permanent. Making this into a one side must win and the other must lose game would be a mistake. The game is still on, and there can be more winners than losers.
I said at the meeting that we couldn't prove someone wouldn't mix Harmony and non-Harmony code and objects, and have non-Harmony code run for-in over a proxy with an iterate trap, and find some kind of surprise.
This is the down side. It is not so obviously terrible and fatal that we all agree to abandon meta-programmability of for-in. It's an unquantified risk, probably small -- greater than the ES5 strict migration tax risks (eval of var, arguments aliasing, caller/callee) and less than the Harmony lexical scope all the way up (no global object on scope chain) risks, in my best guess.
Suggestion: we make old for-in loops never run the proxy iterate trap. Only for-in code that opts into Harmony (analogous to "use strict" changing eval-of-var and arguments) gets the meta-programmability. What say you?
On Nov 22, 2010, at 11:49 AM, Brendan Eich wrote:
On Nov 22, 2010, at 11:14 AM, Oliver Hunt wrote:
On Nov 22, 2010, at 2:08 AM, Tom Van Cutsem wrote:
My arguments in favor of keeping the existing "for-in" syntax and making it meta-programmable:
- Simplicity. Don't underestimate the complexity creep of introducing a new looping construct. Many small changes add up quickly. Especially for novices, having two looping constructs that are so similar will be terribly confusing.
You're not saving the addition of a looping construct, in making for-in behave differently depending on what is on the right hand side of 'in' you're merely adding an additional looping that is not syntactically observable.
Is the additional aspect important enough to split syntax over?
If so, would you make the new form only work on proxies and throw given a non-proxy on the right of "in"?
If not, why not?
Proxies need to have a way to work with for(in) which is the only reason I believe they should be allowed to have a trap for for(in) enumeration. It's necessary if you ever want to have DOM objects like NodeList be defined in terms of proxies.
- Keeping the existing syntax means that the tons of JS code already out there can instantly become client-code to iterators. As Brendan noted: you can use iterators to generate a stream of keys (strings) without confusing clients.
And all existing standards compliant code can no longer rely on for in doing what it has done for years. Suddently for in behaviour may change based on the prototype chain.
That's true in Harmony, and as you noted on the list a while ago, also true in ES5 strict. I decried too much migration tax but allowed we want to break compatibility for important wins. Lexical scope all the way up is one such proposed win, justifying removing the global object from the scope chain.
In my view, letting for-in be reformed by library authors is another case where the migration tax is worth it.
Now, I need to ask whether you are making an absolute statement: Harmony must be runtime as well as syntactically compatible with ES5 strict, i.e., a superset language?
I'm not sure what relevance pure lexical scoping in harmony has to a discussion on the behaviour of for(in). If we're saying that every new API and language feature being discussed for harmony will only be available in harmony, then yes harmony can do whatever it wants, but you've also caused me to lose any interest in implementing harmony at that point. If every feature being discussed is only usable inside harmony code then the migration cost of individual features won't be relevant as it's a distinct language with no impact on ES.
- Client code to proxies that emulate objects with lots of properties can continue to use for-in. It just feels wrong that client code should change because of pure implementation details of an object (i.e. whether it eagerly or lazily generates its keys).
My understanding was that proxies would have a trap to allow them to generate an array of strings to be used.
That's true, the fundamental trap is enumerate, but as noted on the wiki and pointed out by Waldemar when we reviewed proxies and moved them to harmony:proposals status, this does not work well for "large" and "infinite" objects.
We moved proxies to harmony:proposals status with the agreement that the iteration protocol would address such hard cases. The iterators proposal does that, including details about proxies as prototypes of other objects (you don't want to switch to iterate if you start with enumeration -- you must call the proxy handler's enumerate trap).
Last week we agreed toward the end of the meeting, with some lack of clarity about how to do this, to recast for (x in y) as for (x : keys(y)) and allow either enumerate or (if provided) iterate to be used to handle large/infinite objects.
Now the for-: syntax looks like a mistake, and we still haven't reworked things to address the required large/infinite cases.
I don't understand why : was a mistake, the only counter argument i saw was that it didn't work well with type annotations, which i see no value in and aren't being discussed for harmony or es-next. Honestly I wish people would stop treating type annotations as something important -- ES is a dynamically typed language, trying to shoe-horn static typing in seems strange.
I'm reviewing all this because I do not think everyone has kept up with the details. But the details matter, and they have some irreducible complexity we can't wish away. They motivate more than just the enumerate trap which eagerly returns all the property keys
Of course, if an object changes its behavior from iterating keys to iterating values, this breaks clients and they should be modified to use "for (var k in keys(obj))". But I don't see how this differs from any other such changes made to an object. The important thing to note here is that turning an object into an iterator requires explicit action by the programmer. If iterators were implemented ala Python using a magical "iterate" hook, then I'd complain because Harmony code could silently turn normal objects into iterators. But there's no such risk with this proposal.
I think that's a key point worth re-iterating: iterators and regular objects are sufficiently distinct so that there's no risk of automatically converting one into the other. There is only a risk to existing client-code if a Harmony programmer changes a normal object into an iterator. But at that point the programmer knows he's making a non-upwards-compatible change and clients should be changed accordingly. I don't see how the for-in case differs in this respect, fundamentally, from say, renaming a method.
I do not expect the behaviour of for(in) to change from harmony to non-harmony code. Ignoring all other concerns it would mean the behaviour of objects passed from a harmony context to a non-harmony context would be unexpected.
That is a risk but it is not an absolute. It is one end of a trade-off. The other end is the benefit of avoiding new and hard-to-make-winning syntax, splitting for-in and for-: by dynamic type of what is on the right-hand side of in vs. :, complicating the surface language, and precluding library authors from helping in the near term.
How do library authors help? They can't add value enumeration of anything as that will break any existing code that uses for(in) over any of their objects. And given the desire that the library writers have to make their libraries paper over browser differences they're not going to introduce changes that completely change object behaviour from one engine to the next.
If there were a separate syntax, they could provide a custom iterator, and tell devs it was there, so devs would be able to use the nicer enumeration feature when they know that the syntax is available.
If this was going to be the outcome why did we even spend time discussing this?
Because we were trying to reach agreement. Who says failure is not an option? Failure is important. It also is not permanent. Making this into a one side must win and the other must lose game would be a mistake. The game is still on, and there can be more winners than losers.
I said at the meeting that we couldn't prove someone wouldn't mix Harmony and non-Harmony code and objects, and have non-Harmony code run for-in over a proxy with an iterate trap, and find some kind of surprise.
This is the down side. It is not so obviously terrible and fatal that we all agree to abandon meta-programmability of for-in. It's an unquantified risk, probably small -- greater than the ES5 strict migration tax risks (eval of var, arguments aliasing, caller/callee) and less than the Harmony lexical scope all the way up (no global object on scope chain) risks, in my best guess.
Suggestion: we make old for-in loops never run the proxy iterate trap. Only for-in code that opts into Harmony (analogous to "use strict" changing eval-of-var and arguments) gets the meta-programmability. What say you?
I don't believe harmony-only is reasonable, it would mean that for a developer to use just the new enumerator they would have to rewrite everything to deal with the now missing global object. And once again, i think using "harmony" as a mode switch to change behaviour of things unrelated to lexical scoping is the wrong approach.
On Nov 22, 2010, at 12:09 PM, Oliver Hunt wrote:
On Nov 22, 2010, at 11:49 AM, Brendan Eich wrote:
On Nov 22, 2010, at 11:14 AM, Oliver Hunt wrote:
On Nov 22, 2010, at 2:08 AM, Tom Van Cutsem wrote:
My arguments in favor of keeping the existing "for-in" syntax and making it meta-programmable:
- Simplicity. Don't underestimate the complexity creep of introducing a new looping construct. Many small changes add up quickly. Especially for novices, having two looping constructs that are so similar will be terribly confusing.
You're not saving the addition of a looping construct, in making for-in behave differently depending on what is on the right hand side of 'in' you're merely adding an additional looping that is not syntactically observable.
Is the additional aspect important enough to split syntax over?
If so, would you make the new form only work on proxies and throw given a non-proxy on the right of "in"?
If not, why not?
Proxies need to have a way to work with for(in) which is the only reason I believe they should be allowed to have a trap for for(in) enumeration. It's necessary if you ever want to have DOM objects like NodeList be defined in terms of proxies.
Yes, that's already in the harmony:proxies design via the enumerate trap.
You didn't answer my questions, though. If we were to add for (x : y) as you among others seemed to want last week, would it throw on a non-proxy on the right of :?
The answer matters both so we can hope to define for-: (whatever its syntax, if not for-in), and so we can reach agreement on premises. One premise: new users of Harmony implementations can just always use for-: as dherman said. They never have to use for-in. For enumeration, they use for (k in keys(o)). Agree or disagree? If you are indifferent then you're not really participating :-/.
- Keeping the existing syntax means that the tons of JS code already out there can instantly become client-code to iterators. As Brendan noted: you can use iterators to generate a stream of keys (strings) without confusing clients.
And all existing standards compliant code can no longer rely on for in doing what it has done for years. Suddently for in behaviour may change based on the prototype chain.
That's true in Harmony, and as you noted on the list a while ago, also true in ES5 strict. I decried too much migration tax but allowed we want to break compatibility for important wins. Lexical scope all the way up is one such proposed win, justifying removing the global object from the scope chain.
In my view, letting for-in be reformed by library authors is another case where the migration tax is worth it.
Now, I need to ask whether you are making an absolute statement: Harmony must be runtime as well as syntactically compatible with ES5 strict, i.e., a superset language?
I'm not sure what relevance pure lexical scoping in harmony has to a discussion on the behaviour of for(in).
It's a breaking change to the language's runtime semantics, but not to any syntax. It seems entirely analogous to the case of migrating for-in code from pre-Harmony into Harmony, and as I guessed last time, losing the global object looks strictly riskier in terms of unintended global property aliasing breakage.
If we're saying that every new API and language feature being discussed for harmony will only be available in harmony, then yes harmony can do whatever it wants,
We are not saying that because we do not want gratuitous differences for users or implementors. This is pretty clear from all our work. If we really wanted a totally new language, not only would it make for roughly 2x the learning curve for users, and lots of confused-mode bugs, it would make implementors do a bunch more work, approaching 2x.
And as I also argued last week, TC39 would never pull it off. It would be design by committee and it would fail.
So (in case this isn't obvious; I thought it was), we are trying to extend ES5 strict mode with only a few well-chosen runtime semantic shifts. Possibly "few" will be "one": the global object removal in favor of lexical scope.
New syntax brings new semantics of course.
but you've also caused me to lose any interest in implementing harmony at that point.
That's a straw man of your own devising, and I just knocked it down. Can we please get back to the crucial issues?
If every feature being discussed is only usable inside harmony code then the migration cost of individual features won't be relevant as it's a distinct language with no impact on ES.
There's still a migration tax in porting from old to new language, however similar they are. You yourself raised this re: ES5 strict, which changes runtime semantics without any syntactic change (eval-of-var, arguments). It is the same point I'm making about lexical scope removing the global object. I hope it is clear now, both as a risk and an opportunity!
But it was never an opportunity to make a totally new and different language.
- Client code to proxies that emulate objects with lots of properties can continue to use for-in. It just feels wrong that client code should change because of pure implementation details of an object (i.e. whether it eagerly or lazily generates its keys).
My understanding was that proxies would have a trap to allow them to generate an array of strings to be used.
That's true, the fundamental trap is enumerate, but as noted on the wiki and pointed out by Waldemar when we reviewed proxies and moved them to harmony:proposals status, this does not work well for "large" and "infinite" objects.
We moved proxies to harmony:proposals status with the agreement that the iteration protocol would address such hard cases. The iterators proposal does that, including details about proxies as prototypes of other objects (you don't want to switch to iterate if you start with enumeration -- you must call the proxy handler's enumerate trap).
Last week we agreed toward the end of the meeting, with some lack of clarity about how to do this, to recast for (x in y) as for (x : keys(y)) and allow either enumerate or (if provided) iterate to be used to handle large/infinite objects.
Now the for-: syntax looks like a mistake, and we still haven't reworked things to address the required large/infinite cases. I don't understand why : was a mistake, the only counter argument i saw was that it didn't work well with type annotations,
No, you just replied to Tom's arguments against for-: that did not depend on annotation syntax. That is another counter-argument, and you must have read it since you replied directly to his message.
which i see no value in and aren't being discussed for harmony or es-next.
Mark is putting guards up as a candidate proposal.
Honestly I wish people would stop treating type annotations as something important -- ES is a dynamically typed language, trying to shoe-horn static typing in seems strange.
Guards, not types. Mark is right to want to avoid anything connoting static type.
That is a risk but it is not an absolute. It is one end of a trade-off. The other end is the benefit of avoiding new and hard-to-make-winning syntax, splitting for-in and for-: by dynamic type of what is on the right-hand side of in vs. :, complicating the surface language, and precluding library authors from helping in the near term. How do library authors help? They can't add value enumeration of anything as that will break any existing code that uses for(in) over any of their objects.
As Tom pointed out (re-read his message :-|), they can make enumeration work for large/lazy/infinite objects. No non-string non-keys required.
And given the desire that the library writers have to make their libraries paper over browser differences they're not going to introduce changes that completely change object behaviour from one engine to the next.
That's a nice argument for not changing anything, but of course (as I keep saying ad nauseum here, since it is a perniciously selective argument), we are designing for the longer run. ES1 came after original JS; ES2 next; ES3 soon enough; long stretch of de-facto standards; then ES5. Progress happens, and we are planning on it and participating in it, not just giving up or trying to freeze things at some point.
If there were a separate syntax, they could provide a custom iterator, and tell devs it was there, so devs would be able to use the nicer enumeration feature when they know that the syntax is available.
That's not what I meant by "near term", since the near term includes browsers that don't grok the new syntax. The use-cases of large/lazy/infinite proxies exist in all time frames, short and long run.
If this was going to be the outcome why did we even spend time discussing this?
Because we were trying to reach agreement. Who says failure is not an option? Failure is important. It also is not permanent. Making this into a one side must win and the other must lose game would be a mistake. The game is still on, and there can be more winners than losers.
I said at the meeting that we couldn't prove someone wouldn't mix Harmony and non-Harmony code and objects, and have non-Harmony code run for-in over a proxy with an iterate trap, and find some kind of surprise.
This is the down side. It is not so obviously terrible and fatal that we all agree to abandon meta-programmability of for-in. It's an unquantified risk, probably small -- greater than the ES5 strict migration tax risks (eval of var, arguments aliasing, caller/callee) and less than the Harmony lexical scope all the way up (no global object on scope chain) risks, in my best guess.
Suggestion: we make old for-in loops never run the proxy iterate trap. Only for-in code that opts into Harmony (analogous to "use strict" changing eval-of-var and arguments) gets the meta-programmability. What say you?
I don't believe harmony-only is reasonable, it would mean that for a developer to use just the new enumerator they would have to rewrite everything to deal with the now missing global object.
That's not true in general. I can write for (k in keys(o)) ... in Harmony without rewriting everything to avoid keys being redefined if I'm using only code I control or know well.
So why is it an absolute -- why is it not "reasonable", ever, period full stop?
Migrating into Harmony, losing one's dependencies on the global object with early errors from the lexical scope checker built into the JS compiler, could be well worth the trouble.
Likewise, migrating to a meta-programmable for-in could be much better than having to rewrite to use new syntax, teach it to users, implement it, standardize it, wait for it to be deployed, etc.
These are trade-offs, not absolutes.
You write about what you "expect" and "believe", but that's no argument, it's an statement of your expectation or belief. It's circular unless you're arguing from authority.
We have to reason together, debate technical merits and demerits, argue about big picture and little, not lose the forest for the trees. Just saying "no" doesn't lead anywhere, and if you can say "yes" to lexical scope all the way up, then I think you're not being consistent.
And once again, i think using "harmony" as a mode switch to change behaviour of things unrelated to lexical scoping is the wrong approach.
Can you please say why it's "wrong".
Why was lexical scope the only possible change?
I agree that lexical scope all the way up is important, and while I do not share Alex and Arv's optimism that TC39 can design and standardize much more in the way of new semantics for existing or new syntax (what I jokingly called superfunrubypythonjavascript), neither is it obvious that we can only remove the global object and thereafter freeze runtime semantics for all syntax that's in ES5.
From that global object removal a bunch of new issues arise around the standard library and modules. It's not a simple "just this one [big] change" and anyway, we do not have agreement to stop all further evolution there.
Recap:
I proposed meta-programming for-in code that opts into Harmony, to address your specific concern about a proxy with an iterate trap on its handler leaking into pre-Harmony for-in code and being evaluated on the right of "in".
You responded by arguing that we don't want to change runtime semantics apart from removing the global object.
I reply that this is arbitrary and incomplete. We still have a world of trade-offs. We can only get so many right in committee and for the next edition.
Lexical scope is a big trade-off (semantics-breaking change); we seem to agree on that being worthwhile.
The committee already agreed that large/lazy/infinite objects are another, and that needs to be resolved for proxies, which are already in harmony:proposals status.
The solution offered for large/lazy objects (dropping infinite, lazy covers it better) is iterators. There's no counterproposal. The iterators proposal makes for-in metaprogarmmable, which leads directly to the concern about old for-in code being given a proxy.
I've addressed that with a quick "opt-in" solution. I really don't want to goto step 1 under Recap: and iloop. To avoid that, I think it's incumbent on you to give an argument, not just an assertion of belief.
I value your opinion, I'm sorry if I sound mean or pushy. If you don't want to make that argument, someone else who agrees with you may, or we'll just have to soldier on somehow. But since we came this far, I hope you can rally and respond to my query by saying why even opt-in-only for-in metaprogrammability is the "wrong approach".
On Nov 22, 2010, at 12:39 PM, Brendan Eich wrote:
On Nov 22, 2010, at 12:09 PM, Oliver Hunt wrote:
How do library authors help? They can't add value enumeration of anything as that will break any existing code that uses for(in) over any of their objects.
As Tom pointed out (re-read his message :-|), they can make enumeration work for large/lazy/infinite objects. No non-string non-keys required.
The important point here is that for-in won't choke old browsers. New library code in the near term (when Harmony and pre-Harmony impls are in the field), assuming we let for-in be metaprogrammed in Harmony, can object-detect and meta-program for-in, and client and library code can use for-in and it will fail soft or fall back by other means.
Not so if we add new syntax (for-: or anything old browsers will choke on).
On Nov 22, 2010, at 11:49 AM, Brendan Eich wrote:
On Nov 22, 2010, at 11:14 AM, Oliver Hunt wrote:
And all existing standards compliant code can no longer rely on for in doing what it has done for years. Suddently for in behaviour may change based on the prototype chain.
That's true in Harmony,
I think I misread your "based on the prototype chain" words.
I was making the general case for runtime semantic fixes -- few and worth making -- in Harmony as in ES5 strict.
But I think you were supposing a proxy on a non-proxy's prototype chain could start returning non-string non-keys via its iterate trap. Not so. Here's a session with a fresh js shell (Firefox 4 latest code):
js> var proxy = Proxy.create({ has: function () { return false; }, enumerate: function () { return ["a", "b", "c"]; }, iterate: function () { for (let i = 0; i < 10; i++) yield i; } }); js>
js> var obj = Object.create(proxy);
js>
js> for (var i in proxy) print(i); 0 1 2 3 4 5 6 7 8 9 js>
js> for (var i in obj) print(i); a b c
Once for-in starts up a non-proxy object's prototype chain, only enumerate traps -- never iterate. And enumerate is fundamental, so if missing, the for in fails:
js> var proxy = Proxy.create({ has: function () { return false; }, iterate: function () { for (let i = 0; i < 10; i++) yield i; } }); js>
js> var obj = Object.create(proxy);
js>
js> for (var i in proxy) print(i); 0 1 2 3 4 5 6 7 8 9 js>
js> for (var i in obj) print(i); typein:11: TypeError: enumerate is not a function
Rather than adding additional confusion by trying to comment on snippets from this thread, I think I'll just state my current opinions on the subject and the principles that they are based upon:
1a) New semantics should use new syntax in a manner that clearly avoids confusion with existing syntax. 1b) Syntax should generally be suggestive of a reasonable interpretation of the semantic 1c) Harmony is not being designed using the "no new syntax" rule 1d) There is nothing sacred about "for" as the initial keyword of an enumeration statement.
1a) New semantics should use new syntax in a manner that clearly avoids confusion with existing syntax. 1b) Syntax should generally be suggestive of a reasonable interpretation of the semantic 1c) Harmony is not being designed using the "no new syntax" rule 1d) There is nothing sacred about "for" as the initial keyword of an enumeration statement.
Nobody said "sacred" -- I'm not genuflecting. :) Seriously, the reason for using "for" is that it's one of the most stable, common, and universally-used keywords for iteration in almost all languages. Introducing a new initial keyword is straying really far from precedent, both in JS and other imperative languages.
But I appreciate your spelled-out premises. I think my main quibble is with 1a as a rule. I might prefer something like:
2a) If existing syntax is given new semantics, it should extend the existing semantics conservatively. Otherwise, the new semantics should get new syntax.
From these I conclude that new iteration semantics should be syntactically distinct from the current for-in and probably the greater the syntactic distance from for-in the better. Following these principles, here is a suggestion for a new enumeration statement that makes use of existing reserved words:
enum key with keys(x) { alert(key) }
This is clever, but it just seems to go off the deep end: the syntax is too inconsistent with JS precedent. Also, "enum" is the wrong keyword -- in JS parlance, this is "iteration" not "enumeration."
I guess I'm still open to new syntaxes, but I also still feel that when you step back and weigh the trade-offs, the cost of all this new syntax is incommensurate with the amount of new semantics, and moreover the traditional for-in syntax is still the sweetest I've seen for custom iteration. I would rather extend the best syntax and leave the legacy special case as a very small wart than have a warty syntax with a supremely orthogonal semantics.
- Whenever possible, less general pre-existing syntactic forms should be redefined to desugar into new more general forms.
I think this is pretty uncontroversial; whatever syntax we decide on, the specific legacy construct can be defined in terms of the more general new construct.
- Proxy traps should be defined based upon the new, more general semantics not legacy less general semantics.
Define the traps necessary to support enum-with and depend upon the desugaring to take care of legacy for-in.
You don't think for-in should even allow the enumerate trap? This seems to go against the design approach of proxies; it's not just for introducing new meta-programmable constructs, but also for meta-programming existing facilities.
- Provide builtin-library alternatives for new statements that can be used without down-rev syntax errors:
This seems like a good idea.
2a) If existing syntax is given new semantics, it should extend the existing semantics conservatively. Otherwise, the new semantics should get new syntax.
Perhaps I should have numbered that 1a'). :)
On Nov 22, 2010, at 3:17 PM, Allen Wirfs-Brock wrote:
Rather than adding additional confusion by trying to comment on snippets from this thread, I think I'll just state my current opinions on the subject and the principles that they are based upon:
1a) New semantics should use new syntax in a manner that clearly avoids confusion with existing syntax. 1b) Syntax should generally be suggestive of a reasonable interpretation of the semantic 1c) Harmony is not being designed using the "no new syntax" rule 1d) There is nothing sacred about "for" as the initial keyword of an enumeration statement.
I will echo Dave in saying thanks for spelling out principles. Also that "sacred" is a bit one-sided -- I do not think your (1a) is sacred either. Are we "even"? :-|
The forest-for-the-trees meta-point is that if we conservatively add new syntax when meta-programming existing would have gone over without incident, we've made the language irrevocably bigger. We can't ever take back the new syntax or get rid of the old. We will have a long wait switching developers over to the new, which has ripple effects as Tom's post pointed out.
But no need to beat that drum again, I will stifle.
From these I conclude that new iteration semantics should be syntactically distinct from the current for-in and probably the greater the syntactic distance from for-in the better. Following these principles, here is a suggestion for a new enumeration statement that makes use of existing reserved words:
enum key with keys(x) {
This is a clever homage to "with" :-P.
The parenthesis-free head is unusual and invites Go-like innovation elsewhere: will Harmony drop mandatory parens around if, while, etc. heads?
Iteration is not numbering or naming, so taking enum for this purpose promulgates a misnomer and precludes us using enum for a categorical sum declaration of some sort.
- Whenever possible, less general pre-existing syntactic forms should be redefined to desugar into new more general forms.
ES1-5 for-in can be defined via desugaring to enum-with, for example:
//desugar: for (var x in foo) {alert(foo[x])} var x; enum __x with __ES5forinEnumerator(x) { x=__x; {alert(foo[x])} }
This goes against the Proxy design. A proxy shouldn't have to special case __ES5forinEnumerator to customize for-in as opposed to enum-with, and customizing for-in should not affect enum-with when up the proto chain as one of my recent posts showed with a js shell session.
If we must have new syntax to get customized iteration through TC39, we can spend way too much time coming up with acceptable syntax. But before we dive into that fun timekilling process, let's be really sure we are not missing the forest for the trees.
-----Original Message---
-----Original Message---
On Nov 22, 2010, at 5:39 PM, Allen Wirfs-Brock wrote:
AWB: Thanks, I like that. Trying to articulate some principles was the main point. I felt the thread was feeling like it getting hung up on unstated and/or unshared prinicples.
I agree, and I try to make mine clear (let me know where I fail). It's important not only for reaching agreement (backing up and trying a different deductive path forward) but also in rejecting bogus premises, or reprioritizing.
AWB: I worry that it is hard for people to learn when the same name is applied to different concepts. We already have two forms of the "for" statement that people have to learn.
The C-based for(;;) loop is not really relevant in my experience. JS hackers use it (more below on why they might rather not), but they always distinguish it from "for-in".
Some good fraction of JS hackers know Python, where for-in has enumeration like behavior on dictionaries (just like JS objects), but value iteration on lists.
The most frequent (in my hearing) unsatisfied expectation of users of for-in is not that it is a second form after the C-based for(;;) -- there's no mention of for(;;). Rather, it is that for (v in [1,2,3]) fails to iterate 1,2,3, instead enumerating "0","1","2".
We can't satisfy this Pythonic expectation directly, but with meta-programmable for-in, users and library authors can help themselves, using keys, values, and other iterator factory functions.
Further complicating it feels like it would be a pedagogical nightmare.
I think this is exaggerated -- there's really no relation to for(;;), and the first-year students move up and on.
The nightmare of unprogrammable for-in, if I may use that word, is both the non-interoperable mess of for-in today in ES3-ES5-level browsers, plus the enumerate trap of tomorrow -- never mind an iterate trap. But most programmers do not need to know about proxies.
So weigh the nightmares by how many students reach "JS grad school", where they learn about proxies at all.
And give some weight to middling students of the future libraries, who will be able to for-in over array and other array-like objects' values. The C-style for (var i=0; i < a.length; i++) { var v = a[i]; ... } alternative-nightmare would be vanquished by the dawn of for (var v in values(a)) { ... }.
This is all kind of fluffy, on both sides of our argument. Are we really concerned about cognitive load of for (;;) vs. for-in? That ship sailed long ago and I argue it's as likely that meta-programmable for-in heals the rift, as it is that iterate on top of enumerate makes a new rift.
The most frequent (in my hearing) unsatisfied expectation of users of for-in is not that it is a second form after the C-based for(;;) -- there's no mention of for(;;). Rather, it is that for (v in [1,2,3]) fails to iterate 1,2,3, instead enumerating "0","1","2".
We can't satisfy this Pythonic expectation directly, but with meta-programmable for-in, users and library authors can help themselves, using keys, values, and other iterator factory functions.
Library authors can't really do this without providing their own array classes, modifying the array prototype would be too dangerous in a web environment to be done by a library as it would not work with other code that may expect current for-in behaviour, same applies to NodeList's, etc in the DOM. This is another reason i'm in favour of a different syntax - we could make it iterate values as people "expect".
Further complicating it feels like it would be a pedagogical nightmare.
I think this is exaggerated -- there's really no relation to for(;;), and the first-year students move up and on.
The nightmare of unprogrammable for-in, if I may use that word, is both the non-interoperable mess of for-in today in ES3-ES5-level browsers, plus the enumerate trap of tomorrow -- never mind an iterate trap. But most programmers do not need to know about proxies.
I am most concerned about this making for-in much more complicated; Yes python has had meta-programmable objects forever (give or take a few years :D) which shows people can understand the behaviour, but the default behaviour exhibited in python is different, and more in line with what people expect. By changing for(in) behaviour in harmony you're trying to give python like meta-programming but with less than ideal defaults for new developers, and confusion for experienced developers.
I think the other problem i have is that what people really seem to want is value enumeration of arrays (and array-likes), and by re-using the syntax you basically ensure that at the very least arrays (and other array-likes) will never get that behaviour by default. In addition to that it would be very dangerous for any library to override the iterate trap as libraries want to interact well with other libraries, and also provide a consistent development environment across multiple engines and versions. That leads me to expect that libraries would not override the enumerate trap.
If we had a new syntax, in addition to allowing meta-programming from the get-go, we could also provide default enumerate traps for arrays, etc that is inline with what people expect.
On Nov 22, 2010, at 7:07 PM, Oliver Hunt wrote:
The most frequent (in my hearing) unsatisfied expectation of users of for-in is not that it is a second form after the C-based for(;;) -- there's no mention of for(;;). Rather, it is that for (v in [1,2,3]) fails to iterate 1,2,3, instead enumerating "0","1","2".
We can't satisfy this Pythonic expectation directly, but with meta-programmable for-in, users and library authors can help themselves, using keys, values, and other iterator factory functions.
Library authors can't really do this without providing their own array classes, modifying the array prototype would be too dangerous
Modifying the array prototype isn't relevant here (proxies on the prototype do not iterate, they enumerate -- see prior mail with js shell session excerpts). Not sure what you mean if not this -- adding methods to Array.protoype? PrototypeJS does this.
in a web environment to be done by a library as it would not work with other code that may expect current for-in behaviour, same applies to NodeList's, etc in the DOM.
I think you're arguing against something not proposed.
for (var v in values(a)) ...
requires no prototype monkeypatching. Somehow inserting a proxy with a handler having an iterate trap at the front of the Array.prototype chain won't work as noted already. If you thought I meant this:
for (var v in [1,2,3]) print(v);
printing 1 2 3, as I wrote in words cited at the top of this message "We can't satisfy this Pythonic expectation directly". Something like values would need to be called around the [1,2,3].
This is another reason i'm in favour of a different syntax - we could make it iterate values as people "expect".
I'm sure you are in favor of different syntax, but the argument you just made doesn't support that and it seems to have been based on a misunderstanding.
Further complicating it feels like it would be a pedagogical nightmare.
I think this is exaggerated -- there's really no relation to for(;;), and the first-year students move up and on.
The nightmare of unprogrammable for-in, if I may use that word, is both the non-interoperable mess of for-in today in ES3-ES5-level browsers, plus the enumerate trap of tomorrow -- never mind an iterate trap. But most programmers do not need to know about proxies.
I am most concerned about this making for-in much more complicated; Yes python has had meta-programmable objects forever (give or take a few years :D) which shows people can understand the behaviour, but the default behaviour exhibited in python is different, and more in line with what people expect. By changing for(in) behaviour in harmony you're trying to give python like meta-programming but with less than ideal defaults for new developers, and confusion for experienced developers.
Let's not argue by asserting who will be confused. I've already replied to Allen that the sword has two edges, one pointed away from skilled programmers. It's simply selective arguing to say only bad things happen in one direction. You can get cut; you can cut someone not intended. The issue is not whether for-in should be meta-programmable -- Proxy has an enumerate trap -- but how much.
I think the other problem i have is that what people really seem to want is value enumeration of arrays (and array-likes), and by re-using the syntax you basically ensure that at the very least arrays (and other array-likes) will never get that behaviour by default.
If we wanted to change Harmony's runtime semantics to be even less compatible than it is with no global object, we could wave a spec-wand and do that. Why don't we? The reasons won't go away.
The best we can do in any mostly-compatible spec edition is add new forms, whether syntax or API, and see if the cows beat new paths.
Dave's early message then observed that if all the developer cows happily use the new iteration form, the old one is basically dead. Reforming for (var v in [1,2,3]) at that very late date to iterate over values, assuming such a compatibility break is acceptable, simply won't matter by the premise: the cows moved to the new field.
In addition to that it would be very dangerous for any library to override the iterate trap as libraries want to interact well with other libraries,
You can't override the iterate trap. Handlers are stratified and encapsulated. This is basic to the harmony:proxies design.
and also provide a consistent development environment across multiple engines and versions. That leads me to expect that libraries would not override the enumerate trap.
Do you mean wrap a proxy with a different proxy having a different enumerate or iterate trap? Wrapping is not overriding. No mutation, different object identities.
I'm worried we are talking past each other now.
If we had a new syntax, in addition to allowing meta-programming from the get-go, we could also provide default enumerate traps for arrays, etc that is inline with what people expect.
Arrays are not proxies with handlers that contain traps. I think you are imagining unstratified getattr hooks as in Python. But we aren't adding anything like that.
Confused,
On Nov 21, 2010, at 7:05 PM, Brendan Eich wrote:
On Nov 18, 2010, at 4:08 PM, Waldemar Horwat wrote:
Consensus that we should have iterators.
For this, after all these years (JS1.7 added meta-programmable for-in in 2006), I'm grateful, although I wanted to add something your notes did not report:
To get consensus, we made a tentative agreement to leave for-in as it was and not enable any meta-programmability of it, consigning it to the historically-underspecified and non-interoperable enumeration mystery-meat status.
Instead, we did the committee-plays-it-safe thing of inventing new syntax for meta-programmable iteration:
for (var i : x) ...
This is a break from Python and JS1.7+ in SpiderMonkey and Rhino -- although it matches very late Java and C++ extensions that are similar (but not the same), and really not relevant to JS.
Worse, the use of colon in this new for syntax is confusingly similar to long-standing future-proofing intentions around runtime type annotations (aka guards or contracts).
(BTW, I don't think :: is a good type annotation or guard punctuator, btw -- it's the C++ namespace qualification operator, also specified for namespacing by ECMA-357 (E4X) and used that way in ActionScript 3 (and in ES4, RIP). So I was surprised to see :: used for annotation-like syntax in strawman:guards and strawman:trademarks.)
for (var i : x) ... // must be new iteration for (var i : T : x) ... // iteration again, but parsed how? for (var i : T in x) ... // for-in with annotated var
Depending on what T might be, grammatically, this could get sticky for top-down parsers. It is confusing and ugly in any event, IMHO.
Probably we need to take our time and not rush into a meta-programming-here syntax variant of for-in. I'll not propose anything better right now.
If the colon is less future-compatible than we like, why not:
foreach (var i in x)
While "for" is slightly more common for a loop that iterates a container, "foreach" is also quite common, and indeed is often used to name the construct even in languages where the keyword is spelled "for": en.wikipedia.org/wiki/Foreach.
This would allow us to reserve the colon, and also avoid a surprising change to the semantics of the existing for..in construct. It could even DTRT for arrays. Also, it will likely be easier for programmers to talk about the distinction between "for..in loops" and "foreach..in loops" than "for..in loops" and "for..colon" loops.
, Maciej
On Nov 22, 2010, at 11:19 PM, Maciej Stachowiak wrote:
Probably we need to take our time and not rush into a meta-programming-here syntax variant of for-in. I'll not propose anything better right now.
If the colon is less future-compatible than we like, why not:
foreach (var i in x)
All the new words are not reserved, so they could begin a function call expression in extant code:
hi = "there" foreach (i in x) print(i)
means
hi = "there"; foreach(i in x); print(i);
today. Same if you s/foreach/iterate/ or any non-reserved identifier. Wherefore Allen's co-opting of enum (plus with, a decent preposition given enum but 8 chars in two keywords hurt kittens everywhere).
Anyway, the bikeshed is secondary. We need to agree on what meta-programmable for-in means with the enumerate trap (specified by the wiki pages on harmony:proxies), how that changes with the optional iterate trap (strawman:iterators), and when it might matter (for all for-in loops, or only those in Harmony code?).
On Nov 22, 2010, at 11:35 PM, Brendan Eich wrote:
On Nov 22, 2010, at 11:19 PM, Maciej Stachowiak wrote:
Probably we need to take our time and not rush into a meta-programming-here syntax variant of for-in. I'll not propose anything better right now.
If the colon is less future-compatible than we like, why not:
foreach (var i in x)
All the new words are not reserved, so they could begin a function call expression in extant code:
hi = "there" foreach (i in x) print(i)
means
hi = "there"; foreach(i in x); print(i);
today. Same if you s/foreach/iterate/ or any non-reserved identifier. Wherefore Allen's co-opting of enum (plus with, a decent preposition given enum but 8 chars in two keywords hurt kittens everywhere).
One possibility is to add a space as apparently a few other languages do:
for each (var i in x)
Should be unambiguously parsable and easy to understand.
Anyway, the bikeshed is secondary. We need to agree on what meta-programmable for-in means with the enumerate trap (specified by the wiki pages on harmony:proxies), how that changes with the optional iterate trap (strawman:iterators), and when it might matter (for all for-in loops, or only those in Harmony code?).
Fair enough. Another important question is whether for..in should continue to guarantee that all values enumerated would be strings. Preserving that guarantee clearly makes for..in useless for the strawman:iterators purpose, but not for the harmony:proxies enumeration trap (and thus iterators would need new syntax to be usable). Breaking the guarantee may cause problems for existing code. Breaking the guarantee only in Harmony mode creates additional migration tax. Thus, iterators create a potential motive for new syntax that the proxy enumeration trap does not.
Given this, it is useful to identify candidate syntax that doesn't have perceived showstopper problems, so that we can evaluate the tradeoffs among the three prongs of the trilemma.
(As an even more concrete previously mentioned example, a new syntax allows sane defaults for arrays without breaking compatibility for the existing construct.)
, Maciej
On Nov 23, 2010, at 12:11 AM, Maciej Stachowiak wrote:
On Nov 22, 2010, at 11:35 PM, Brendan Eich wrote:
On Nov 22, 2010, at 11:19 PM, Maciej Stachowiak wrote:
Probably we need to take our time and not rush into a meta-programming-here syntax variant of for-in. I'll not propose anything better right now.
If the colon is less future-compatible than we like, why not:
foreach (var i in x)
All the new words are not reserved, so they could begin a function call expression in extant code:
hi = "there" foreach (i in x) print(i)
means
hi = "there"; foreach(i in x); print(i);
today. Same if you s/foreach/iterate/ or any non-reserved identifier. Wherefore Allen's co-opting of enum (plus with, a decent preposition given enum but 8 chars in two keywords hurt kittens everywhere).
One possibility is to add a space as apparently a few other languages do:
for each (var i in x)
Should be unambiguously parsable and easy to understand.
Anyway, the bikeshed is secondary. We need to agree on what meta-programmable for-in means with the enumerate trap (specified by the wiki pages on harmony:proxies), how that changes with the optional iterate trap (strawman:iterators), and when it might matter (for all for-in loops, or only those in Harmony code?).
Fair enough. Another important question is whether for..in should continue to guarantee that all values enumerated would be strings. Preserving that guarantee clearly makes for..in useless for the strawman:iterators purpose,
Why? The primary reason for the iterator protocol is avoiding eager instantiation of the entire result set. If we force stringification in for-in when used with an iterator, the protocol is still alive and well and useful.
We could avoid stringification if next() is called on iterators directly, or provide an additional distinct new for-each or whatever syntax that doesn't stringify.
One possibility is to add a space as apparently a few other languages do:
for each (var i in x)
Should be unambiguously parsable and easy to understand.
This has come up several times. For one, it's incompatible with ECMA-357 (E4X), which is implemented in SpiderMonkey and Rhino. But more importantly, IMO, the syntactic distinction between "for" and "for each" is not at all evocative of their semantic distinction. Indeed, "for" implicitly means "for each" already!
Fair enough. Another important question is whether for..in should continue to guarantee that all values enumerated would be strings.
A fair question, although as I believe Andreas pointed out earlier in the thread, it's subsumed by the more general question of whether for..in should continue to guarantee that all values enumerated would be keys.
Preserving that guarantee clearly makes for..in useless for the strawman:iterators purpose,
Why? The primary reason for the iterator protocol is avoiding eager instantiation of the entire result set. If we force stringification in for-in when used with an iterator, the protocol is still alive and well and useful.
We could avoid stringification if next() is called on iterators directly, or provide an additional distinct new for-each or whatever syntax that doesn't stringify.
IOW, we could usefully have:
-
only one for-in form, that is meta-programmable via iterate(), and that does not guarantee it only iterates over keys
-
two different forms: a) for-in, which is meta-programmable but where the semantics enforces the invariant that all values iterated over are keys; and b) another form (syntax TBD) which is meta-programmable but where the semantics does not modify the values iterated over
Most of this discussion has been about #1 vs. #2, but it's good to point out that 2a is still compatible with a meta-programmable API.
On 2010-11-22, at 02:37, David Herman wrote:
if we allowed for-in to be overloaded, I would tell people that they should deprecate the legacy for-in and replace it with an explicit iterator such as:
for (x in keys(obj))
I have learned a mnemonic for for-in: that it is iterating using the in operator. You propose that I unlearn that? Or in your new hypothetical world does the in operator also get overloaded?
Ramdom thought: Can I use destructuring in for-in?
for ({key:value} in enumerable)
for ([value] in iterable)
On Nov 23, 2010, at 12:11 AM, Maciej Stachowiak wrote:
One possibility is to add a space as apparently a few other languages do:
for each (var i in x)
Should be unambiguously parsable and easy to understand.
This is what ECMA-357, E4X, does, and it indeed iterates values not keys. The obvious criticism is that nothing except (dubious) history says for-in == keys while for-each-in == values. Other symmetries break down:
for (i in o) assert(i in o)
lacks this parallel construct:
for each (v in o) assert(o[what-was-v's-name-again?] === v)
JS1.7 allowed key-value destructuring
for ([k, v] in o) assert(o[k] === v) // assume idemptotent has/get
but this broke a different symmetry:
for ([s, v, o] in tripledb) ...
which wants other destructuring patterns to destructure the value returned by the iterator. Making the two-element-array destructuring pattern mean key-value destructuring frustrates people who want to iterate over a pairdb. So we fixed this in JS1.8, and dherman's iterator proposal has
for (k in keys(o)) ... for (v in values(o)) ... for ([k, v] in properties(o)) ...
EIBTI once again.
Anyway, the bikeshed is secondary. We need to agree on what meta-programmable for-in means with the enumerate trap (specified by the wiki pages on harmony:proxies), how that changes with the optional iterate trap (strawman:iterators), and when it might matter (for all for-in loops, or only those in Harmony code?).
Fair enough. Another important question is whether for..in should continue to guarantee that all values enumerated would be strings. Preserving that guarantee clearly makes for..in useless for the strawman:iterators purpose,
Not quite, as Andreas already argued and (I see, reading ahead) reiterates with dherman adding emphasis.
Stringifying is not good for iterating arbitrary values, in particular property values or key-value pairs, of course. But if you insist (I don't) that for-in always stringify, forever, there are still lazy use-cases for the proposed iterate trap.
but not for the harmony:proxies enumeration trap (and thus iterators would need new syntax to be usable). Breaking the guarantee may cause problems for existing code. Breaking the guarantee only in Harmony mode creates additional migration tax. Thus, iterators create a potential motive for new syntax that the proxy enumeration trap does not.
We're going in circles. Clearly, we could add new syntax. It's also conceivable that we could extend for-in. Trade-offs abound and data on usability hazards is absent in our experience (JS1.7 and up).
Given this, it is useful to identify candidate syntax that doesn't have perceived showstopper problems, so that we can evaluate the tradeoffs among the three prongs of the trilemma.
It's a problem if showstopper criteria are not shared in a committee. Then anyone can stop the show based on divergent principles or unarticulated beliefs.
It therefore seems more important to achieve consensus on principles, and less important to force new syntax as a compromise that may be founded on bad committee game theory.
(As an even more concrete previously mentioned example, a new syntax allows sane defaults for arrays without breaking compatibility for the existing construct.)
Mozilla-specific JS hackers do use
for each (v in a) ...
to iterate values in an array |a|. This is better than
for (i in a) { v = a[i]; ... }
but it's not so much better that TC39 members are happy to adopt for-each-in from ECMA-357. Maybe we will, in the end. It is overlong and "each" is redundant or ambiguous (take your pick), but at this point it beats the rushed for-: ugliness of last week.
On Nov 23, 2010, at 5:19 AM, P T Withington wrote:
On 2010-11-22, at 02:37, David Herman wrote:
if we allowed for-in to be overloaded, I would tell people that they should deprecate the legacy for-in and replace it with an explicit iterator such as:
for (x in keys(obj))
I have learned a mnemonic for for-in: that it is iterating using the
inoperator. You propose that I unlearn that? Or in your new hypothetical world does theinoperator also get overloaded?
Excellent question. One (Java extension language) answer:
www.cs.cornell.edu/Projects/jmatch
Python allows unstratified meta-programming of both its for-in loop/comprehension syntax and its |in| operator.
Harmony Proxies allow meta-programming of |in| already, via the |has| trap. So the answer to your quesiton "does the in operator also get overloaded?" is "Yes, but you have to write two traps, iterate and has".
Ramdom thought: Can I use destructuring in for-in?
for ({key:value} in enumerable)
for ([value] in iterable)
Absolutely. Destructuring (separate proposal but composes well) applies to all LHS and binding forms.
On Nov 23, 2010, at 11:12 AM, Brendan Eich wrote:
Given this, it is useful to identify candidate syntax that doesn't have perceived showstopper problems, so that we can evaluate the tradeoffs among the three prongs of the trilemma.
It's a problem if showstopper criteria are not shared in a committee. Then anyone can stop the show based on divergent principles or unarticulated beliefs.
It therefore seems more important to achieve consensus on principles, and less important to force new syntax as a compromise that may be founded on bad committee game theory.
The principle could be refined from Dave's restatement of Allen's (1a):
1a'. If existing syntax is given new semantics, it should extend the existing semantics conservatively. Otherwise, the new semantics should get new syntax.
But "conservatively" needs defining.
I think rather than try to nail down an abstract principle, we should look at the particular things in themselves. Tom's post did this:
"I think that's a key point worth re-iterating: iterators and regular objects are sufficiently distinct so that there's no risk of automatically converting one into the other. There is only a risk to existing client-code if a Harmony programmer changes a normal object into an iterator. But at that point the programmer knows he's making a non-upwards-compatible change and clients should be changed accordingly. I don't see how the for-in case differs in this respect, fundamentally, from say, renaming a method."
[esdiscuss/2010-November/012208]
This is an argument about APIs. If an object exposed by an API, say returned from an API function, was a plain old object with keys to enumerate, but now (with Harmony implementations targeted) returns a proxy with an iterate trap, and the iterator returns non-strings non-keys (but key is ill-defined if not "string"), then client code could break.
But this is an intentional API change. It is analogous to renaming a method, and plausibly, object detection if not renaming would be used to deal with downrev clients.
What about unintended consequences? An accidental proxy leak from Harmony code to pre-Harmony code is possible. This is why I raised the idea of runtime semantics under Harmony opt-in being required to meta-program for-in. The same opt-in is required to remove the global object from the scope chain in favor of a lexical top frame.
We can (and have!) argue about limits on migration taxation, implementor burdens, etc. But as ES5 strict changed runtime semantics, it seems Harmony will too (lexical scope). So it is not absolutely inconceivable, or even "obviously wrong", that for-in might become metaprogrammable based on its static code "in Harmony" property.
(As an even more concrete previously mentioned example, a new syntax allows sane defaults for arrays without breaking compatibility for the existing construct.)
Mozilla-specific JS hackers do use
for each (v in a) ...
to iterate values in an array |a|. This is better than
for (i in a) { v = a[i]; ... }
but it's not so much better that TC39 members are happy to adopt for-each-in from ECMA-357. Maybe we will, in the end. It is overlong and "each" is redundant or ambiguous (take your pick), but at this point it beats the rushed for-: ugliness of last week.
Lest anyone think I'm throwing in the towel -- I'm not. We need better principles and detailed arguments justifying them or we won't have lasting and sound consensus to do much of anything with the language.
Proxies moved to harmony:proposals status because they were designed carefully, very well specified, and prototype implemented (and the prototyping was essential to completing the spec and getting the design right). On one point of contention in the July meeting, I urged TC39 to let the champions of the proposal (Mark and Tom), decide what to do.
I'd like to continue this champion-decides approach. Committees are terrible at designing, specifying, and implementing. It would be much better if we had multiple Proxy implementations (one for v8 is on github, Tom noted recently: brickysam26/node-proxy), including the iterate trap as proposed, and then some early adopters using proxies and beating on the iteration protocol.
At that point we would have more information on the usability issues. In the mean time, we can try to refine our principles based on concrete analysis. Just making abstract "never do X" rules won't cut it.
On 2010-11-23, at 14:14, Brendan Eich wrote:
On Nov 23, 2010, at 5:19 AM, P T Withington wrote:
On 2010-11-22, at 02:37, David Herman wrote:
if we allowed for-in to be overloaded, I would tell people that they should deprecate the legacy for-in and replace it with an explicit iterator such as:
for (x in keys(obj))
I have learned a mnemonic for for-in: that it is iterating using the
inoperator. You propose that I unlearn that? Or in your new hypothetical world does theinoperator also get overloaded?Excellent question. One (Java extension language) answer:
www.cs.cornell.edu/Projects/jmatch
Python allows unstratified meta-programming of both its for-in loop/comprehension syntax and its |in| operator.
Harmony Proxies allow meta-programming of |in| already, via the |has| trap. So the answer to your quesiton "does the
inoperator also get overloaded?" is "Yes, but you have to write two traps, iterate and has".
How does the in in for-in decide which of it's overloaded meanings applies? Based on the type of the operand? Based on the existence of the enumerate or iterate trap on the operand?
And despite the in in for-in either enumerating or iterating, the in operator only has a single associated trap. The non-parallelism is bugging me.
Ramdom thought: Can I use destructuring in for-in?
for ({key:value} in enumerable)
for ([value] in iterable)
Absolutely. Destructuring (separate proposal but composes well) applies to all LHS and binding forms.
I was being too subtle. I was suggesting something like your JS 1.7 example, where the 'top-level' destructuring is a pattern for the in operation. {key: value} means I want the property keys and their values, [value] means I want the values, and key is (backward-compatible) shorthand for {key: _}. Destructuring iteration over an hash of triples:
for ({key: [s, v, o]} in tripledb) ...
Too cute?
On 11/23/10 11:12, Brendan Eich wrote:
for (k in keys(o)) ... for (v in values(o)) ... for ([k, v] in properties(o)) ...
What are "keys", "values", and "properties" here? Global functions? How would a new object abstraction T customize them just for instances of T?
Waldemarfor (k in keys(o)) ... for (v in values(o)) ... for ([k, v] in properties(o)) ...
What are "keys", "values", and "properties" here? Global functions?
Those are API's suggested in the strawman:iterators proposal. They would be importable from a standard module.
How would a new object abstraction T customize them just for instances of T?
By writing its own custom iteration protocol via proxies with the iterate() trap implemented appropriately. E.g.:
function MyCollection() { }
MyCollection.prototype = {
iterator: function() {
var self = this;
return Proxy.create({
iterate: function() { ... self ... },
...
});
}
}On Nov 23, 2010, at 12:14 PM, P T Withington wrote:
Harmony Proxies allow meta-programming of |in| already, via the |has| trap. So the answer to your quesiton "does the
inoperator also get overloaded?" is "Yes, but you have to write two traps, iterate and has".How does the
inin for-in decide which of it's overloaded meanings applies? Based on the type of the operand? Based on the existence of the enumerate or iterate trap on the operand?
A proxy's handler is a single object with trap methods. These traps work together to create an abstraction. It should be a coherent abstraction (ideally, don't you think? ;-).
If you're implementing an iterator, then the iterate and has traps are enough to make for-in and in agree. From Python experience, just iterate is enough. (JMatch is cute because it can synthesize both iteration and membership testing from Boolean formulae, but we're not going to try for anything like it.)
If you're implementing a container, then questions of mutability arise. Immutable container proxies could have iterate, has, and get. Mutable containers might rather vend iterators via specific API methods, where the iterators are separate proxies that iterate over snapshots. There are lots of possibilities.
And despite the
inin for-in either enumerating or iterating, theinoperator only has a single associated trap. The non-parallelism is bugging me.
There are at least two reasons for this:
- Large and lazy objects cannot afford to enumerate all keys eagerly, as the enumerate trap wants.
- "in" maps to the "has" trap, which must eagerly compute a boolean result.
With just harmony:proxies as proposed, (1) motivates adding a derived trap, call it iterate, that can be used instead of enumerate. This is the proposed solution to the large/lazy object problem that we agreed to fix when moving Proxies to harmony:proposals status.
Notice how "in" does not have a large/lazy problem. Membership queries must be answered eagerly, all at once, with a boolean result. This is (2).
Another reason (3) is to support iterators as a special kind of proxy. All objects can be enumerated if not iterated, but not all objects are iterators. Adding a derived, lazy-iteration trap to use instead of enumerate allows proxies to implement iterators for arbitrary value streams. This is something we propose separately from proxies, but build on proxies (rather than reinventing obvious wheels). We thus handle large/lazy object enumeration (string typed key streams) as well as custom iterateion (arbitrarily typed value streams).
The particulars matter. Tom invokes Einstein's famous "as simple as possible, but not simpler" dictum in his talks on proxies. Both JS's particulars, and hard realities to-do with scaling to large or lazily built objects, motivate non-parallel trap structure here.
Trying to oversimplify will just make an unusably simplistic metaprogramming API that falls down on some use-cases, or too easily leads to incoherent abstractions.
Ramdom thought: Can I use destructuring in for-in?
for ({key:value} in enumerable)
for ([value] in iterable)
Absolutely. Destructuring (separate proposal but composes well) applies to all LHS and binding forms.
I was being too subtle. I was suggesting something like your JS 1.7 example, where the 'top-level' destructuring is a pattern for the
inoperation.{key: value}means I want the property keys and their values,
No, that destructuring pattern {key: value} captures only the "key" property's value, bound to the name "value" on each turn of the loop.
Making it a special form just breaks destructuring for no good reason.
[value]means I want the values,
No, that means get the "0" property of the iterated value and bind it to the name "value".
and
keyis (backward-compatible) shorthand for{key: _}. Destructuring iteration over an hash of triples:for ({key: [s, v, o]} in tripledb) ...
Too cute?
Too restrictive and verbose, also does violence to destructuring by reinterpreting its syntax.
Destructuring lets you pull properties from objects using object and array patterns modeled on initialisers, in a way that honestly desugars to property references and assignments.
Iteration (whatever the syntax) lets you visit a sequence of values, whatever their type and deeper structure.
Destructuring the iterated value conveniently binds values from the deeper structure to names or local variables declared in the iteration's loop syntax and used in the loop body.
Thus there is no good reason to hardcode "key" as a name, or any particular one- or two-property pattern, in order to deal with objects as key/value stores using destructuring and iteration. We can leave destructuring unaltered and provide different iterators for the key, value, and key-value (and triple, quad, etc.) use cases.
As dherman noted, iteration needs a meta-object protocol because neither 2 nor 200 iterator flavors are enough. This is a library issue, but right now libraries have no ability to extend iteration syntax of any kind, and instead must resort to closure-based functional iteration.
Functional iteration certainly works, but it has its verbosity and closure-related runtime costs, and to many users familiar with nearby languages that have nice iteration syntax and metaprogrammability, the lack of those in JS cries out for usable syntax, which in the for-in form is rotting due to enumeration's messiness and underspecification.
In the end, whatever we do with syntax (we agreed to add meta-programmable iteration at last week's meeting, details to be worked out over time), destructuring should compose without special cases.
How would a new object abstraction T customize them just for instances of T?
By writing its own custom iteration protocol via proxies with the iterate() trap implemented appropriately. E.g.:
function MyCollection() { } MyCollection.prototype = { iterator: function() { var self = this; return Proxy.create({ iterate: function() { ... self ... }, ... }); } }
I left out the last step: clients would then use this via:
var coll = new MyCollection();
...
for (var x in coll.iterator()) {
...
}On Nov 23, 2010, at 1:05 PM, Brendan Eich wrote:
If you're implementing a container, then questions of mutability arise. Immutable container proxies could have iterate, has, and get. Mutable containers might rather vend iterators via specific API methods, where the iterators are separate proxies that iterate over snapshots. There are lots of possibilities.
For some reason I typed "container" where I meant "collection" -- could be that both terms work, but collection is better.
On 11/23/10 13:05, David Herman wrote:
How would a new object abstraction T customize them just for instances of T?
By writing its own custom iteration protocol via proxies with the iterate() trap implemented appropriately. E.g.:
function MyCollection() { } MyCollection.prototype = { iterator: function() { var self = this; return Proxy.create({ iterate: function() { ... self ... }, ... }); } }I left out the last step: clients would then use this via:
var coll = new MyCollection(); ... for (var x in coll.iterator()) { ... }Dave
That misses the point of my question. Sure you can define the meaning of for (k in MyNewFunction(o)) or for (k in o.MyMethod())
However, I asked how a new object abstraction T would customize "keys", "values", and "properties" just for instances of T so that instances of T could be usable under the standard pattern:
for (k in keys(o)) ...
for (v in values(o)) ...
for ([k, v] in properties(o)) ...
WaldemarI'd like to contribute what I think are three not-yet-voiced arguments to this discussion:
- Iterators != Collections First, let me briefly restate what I preceive to be the main point of contention: if we overload for-in, then client code that previously executed "for (x in obj)" with obj being a regular object, and now finds itself in Harmony-land where obj is changed to an iterator, then "for (x in obj)" may confuse the client, if the iterator produces non-string values.
Coming back to my point that regular objects don't just become iterator objects overnight: it was not yet emphasized that in Dave's iterators API, iterators are essentially considered as proxies for collections, usually even constructed for one-off iteration use only. That is: the original collection object itself is not iterable, and that means that clients that hold on to just the collection object can never become "for-in"-confused.
I feel that this tremendously reduces the risk of confusion, even though it is still possible, and I have no empirical evidence to back this up. However, reasoning by analogy: in pre 1.5 Java, one also asked a Collection for an Iterator, then iterated using the Iterator, not the original Collection. Of course, Java's static type system virtually rules out all confusion since Iterator is not a subtype of Collection, but I would argue that in this case, it was the idiom more than the type system that avoided confusion. Java programmers didn't get confused because it was idiomatic to write: "for (Iterator it = coll.iterator(); it.hasNext(); ) { ... it.next() ... }"
In Harmony, as Dave has shown, it would be idiomatic to use iterators as follows: "for (var x in coll.iterator()) { ... }" Note: if a harmony programmer would, in a stroke of forgetfulness, forget the ".iterator()" part, he'll get the usual for-in enumeration behavior for 'coll', since 'coll' is a normal collection object, not an iterator. Of course, if the programmer meant to do iteration, this is a bug. But, more importantly, if the programmer wasn't forgetful and he meant what he wrote (i.e. enumerate the keys of the collection object), it'll do the right thing.
If this is the idiom around which Harmony iterators are based, I would find it unlikely (drawing from my experience with Java Iterators) that Harmony programmers will start passing out references to coll.iterator() to clients where they meant to pass 'coll'. I would only expect to see .iterator() in or close to a for-in loop that immediately 'uses up' the iterator.
-
Avoid Iterators that pretend to be Collections Now, in light of the above, one reservation I have about Iterators is the API method called Iterator.for(coll), which creates and returns an iterator that is also a forwarding proxy to coll. This method gives an incentive to users to pass this proxy where they could have also passed 'coll' to a client. I think this is a mistake. Iterators should be distinct from collections, as in Java. If iterators are indeed substitutable for collection objects, then the chance of confusing clients becomes much greater than it should be, effectively undermining argument 1)
-
Who is to blame for client confusion Andreas, in the thread on guard syntax, observed that there is a very thin line between a proxy enumerating duplicate/nonsensical keys (which is possible using just the enumerate() trap, and perhaps even necessary to emulate quirky host objects) and producing arbitrary values. Clients may get confused either way.
It seems to me that we attribute different "weights" to these confusions based on the perceived use of proxies. Why is this? Here's my take on it:
-
proxies that use enumerate() to generate faux and/or duplicate keys are considered by us to somehow "abuse" the trap. We cannot prevent anyone from writing bad proxies, but we also don't expect code that does this to be idiomatic, or part of a popular library. Hence, even though these proxies may deliberately confuse client code, we expect the consequences to be "contained". It's obvious who to blame for the confusion in this case.
-
proxies that use iterate() to generate streams of arbitrary values are considered a legitimate use case. Under the hypothesis that strawman:iterators would become part of the language, it is our expectation that such proxies will become widespread. When they do, client code that previously executed "for (x in obj)" might now find obj bound to an iterator and get confused. Hence, even though it wasn't the intention of the iterator proxy to confuse the client, the confusion happened anyway. This is considered more problematic, because it's hard to figure out who to blame. Both client and proxy meant to cooperate, but failed.
In short:
- Idiomatic use of iterators should prevent client code confusion in the common case.
- Objects don't become iterators overnight. When porting ES5 code to Harmony, I would expect ES5 objects used as collections to now get an "iterator()" method that returns a distinct iterator object. It would be weird for the collection object itself to become an iterator. The iterators API, without Iterator.for, makes the first option the path of least resistance, by far.
Cheers, Tom
2010/11/23 Waldemar Horwat <waldemar at google.com>
Third day rough meeting notes.
instanceof trap for proxies: Withdrawn. Agreed to drop it for now.
Proxy default forwarding handler. Agreed that there should be an easy way to delegate to a default forwarding handler. Waldemar: Why not use prototypes to do this? MarkM, others: Would work, but need to write a proposal for it.
MarkM: Arguing for simplicity (understandability) of the modules proposal
Discussed the merits of modules, asynchrony, scoping, and evaluation phasing. Agreed that static resolution of bindings should produce results indistinguishable from later dynamic resolution of those bindings. Thus, for example it's not ok to asynchronously import * into an outer lexical scope. Also need to ensure that asynchronously loading multiple scripts on a page doesn't violate this rule for the outer lexical scopes, perhaps by making the "global" scope no longer be reflected as an object.
DaveH: Would like to see Harmony lexically scoped all the way up.
Discussion of IEEE 754r decimal cohorts.
Debate over whether for-in should be conceptually restricted to iterating only keys or whether users should be able to do things like: for (var val = values(x) Other points of contention: Iterating keys of an array should return numbers or strings? What should be the syntax for general iteration? Provide a different syntax for user-defined iteration? for (var key : iterator)? Provide a way to iterate only the shallow enumerable properties of an object?
Consensus that we should have iterators.
Generators: Need to have a distinctive syntax in the preamble to distinguish them from functions. Finally inside generators is also an issue because it might not run. Waldemar: don't allow yield inside a try.
Binary data: We have array buffers (not on wiki) and the binary data spec on wiki. Array buffers allow accidental aliasing and endian dependence, and we'd like to not have them.
Two very distinct use cases: WebGL arrays: endian-ignorant, fixed size, preallocated File parsing and other such I/O: need to explicitly specify endianness, dynamically construct representation, deal with variable-sized items such as strings. Endianness is a property of the i/o routines, not of the underlying data buffer.
An arraybuffer is like a void* -- it's a container for view data structures to index into, where those other data structures impose a structure on a slice of the arraybuffer. DaveH: Would be good to enforce no aliasing of views into arraybuffers.
Debate over what primitives should be supported:
Waldemar: In-memory contiguous buffers that allow pointers to simple things such as strings. Example with three congiguous records: int int char* -> "Hello"
int int char* -> "World"
int int char* -> "\n"
Each record is fixed-size and has a fixed contiguous structure but whose layout and exact byte size is otherwise invisible to the ECMAScript programmer. The records contain pointers to the strings, but the strings themselves are stored somewhere else on the heap. There should be no way to read/write these into a file except by explicitly describing how it's serialized or deserialized. Of course, all this could also be done using traditional ECMAScript arrays and objects but at a much higher cost when the data is things like graphics data.
DaveH: An in-memory array of bytes (like an arraybuffer) together with ways to map views onto it to interpret things like ints, strings, or offset-pointers in some external data format (such as a JPEG).
Waldemar's objections to providing views into an in-memory array of bytes:
Turns out that the GL interface doesn't let ECMAScript write directly into GPU memory. The methods make copies as needed.
More discussion of data buffers.
Third day rough meeting notes. Waldemar instanceof trap for proxies: Withdrawn. Agreed to drop it for now. Proxy default forwarding handler. Agreed that there should be an easy way to delegate to a default forwarding handler. Waldemar: Why not use prototypes to do this? MarkM, others: Would work, but need to write a proposal for it. MarkM: Arguing for simplicity (understandability) of the modules proposal Discussed the merits of modules, asynchrony, scoping, and evaluation phasing. Agreed that static resolution of bindings should produce results indistinguishable from later dynamic resolution of those bindings. Thus, for example it's not ok to asynchronously import * into an outer lexical scope. Also need to ensure that asynchronously loading multiple scripts on a page doesn't violate this rule for the outer lexical scopes, perhaps by making the "global" scope no longer be reflected as an object. DaveH: Would like to see Harmony lexically scoped all the way up. Discussion of IEEE 754r decimal cohorts. Debate over whether for-in should be conceptually restricted to iterating only keys or whether users should be able to do things like: for (var val = values(x) Other points of contention: Iterating keys of an array should return numbers or strings? What should be the syntax for general iteration? Provide a different syntax for user-defined iteration? for (var key : iterator)? Provide a way to iterate only the shallow enumerable properties of an object? Consensus that we should have iterators. Generators: Need to have a distinctive syntax in the preamble to distinguish them from functions. Finally inside generators is also an issue because it might not run. Waldemar: don't allow yield inside a try. Binary data: We have array buffers (not on wiki) and the binary data spec on wiki. Array buffers allow accidental aliasing and endian dependence, and we'd like to not have them. Two very distinct use cases: WebGL arrays: endian-ignorant, fixed size, preallocated File parsing and other such I/O: need to explicitly specify endianness, dynamically construct representation, deal with variable-sized items such as strings. Endianness is a property of the i/o routines, not of the underlying data buffer. An arraybuffer is like a void* -- it's a container for view data structures to index into, where those other data structures impose a structure on a slice of the arraybuffer. DaveH: Would be good to enforce no aliasing of views into arraybuffers. Debate over what primitives should be supported: Waldemar: In-memory contiguous buffers that allow pointers to simple things such as strings. Example with three congiguous records: int int char* -> "Hello" int int char* -> "World" int int char* -> "\n" Each record is fixed-size and has a fixed contiguous structure but whose layout and exact byte size is otherwise invisible to the ECMAScript programmer. The records contain pointers to the strings, but the strings themselves are stored somewhere else on the heap. There should be no way to read/write these into a file except by explicitly describing how it's serialized or deserialized. Of course, all this could also be done using traditional ECMAScript arrays and objects but at a much higher cost when the data is things like graphics data. DaveH: An in-memory array of bytes (like an arraybuffer) together with ways to map views onto it to interpret things like ints, strings, or offset-pointers in some external data format (such as a JPEG). Waldemar's objections to providing views into an in-memory array of bytes: - There is a huge menagerie of basic data types used in files: Many different kinds of string termination, many different kinds of offsets and record counts, computed lengths, etc. - This structure is not appropriate for general modifications. An ECMAScript program cannot practically insert something into the middle without adjusting offsets, lengths, etc. in the block records around the inserted data. Instead, the offsets, lengths, etc. should be consumed by deserialization and generated by serialization. Turns out that the GL interface doesn't let ECMAScript write directly into GPU memory. The methods make copies as needed. More discussion of data buffers.