Comprehensions, Where Art Thou?
I can give it a go ;)
We can start with generator comprehensions, which are a specific way of constructing lazy sequences. It's a syntactic feature, which has a cost on the language level -- makes you wonder, can this be done without syntax?
Turns out, yes: you can do it by constructing a graph of combinators, with normal method syntax. So you can do:
var sequence = iterable.lazy().if(somePredicate)
where lazy and if might be:
Iterable.prototype.lazy = function* () {
for (var x of this) yield x;
}
Iterable.prototype.if = function* (pred) {
for (var x of this) if pred(x) yield x;
}
The neat thing about this is that it's extensible -- it privileges the users to the same degree as the language designers. If there's a new pipeline-like operator, it can be polyfilled on Iterator.prototype.
It also allows for look-alike APIs for other kinds of traversal: for example, the same syntax could map over parallel arrays in parallel.
There is also an argument that a more combinator-like syntax can better support pushing values into iterators, a concept described as "Observables" rather than "Iterables", but I don't remember the details and can't describe it adequately. The end point of all these arguments is that a more combinator-like, linq-like API is as expressive and more extensible than a hard-coded syntactic facility.
I love array comprehensions -- they are light-weight both from a performance and syntactic point of view. You can quickly build up short arrays, and at least in SpiderMonkey you get a lovely ionified loop. Generator comprehensions were unlikely to be as light-weight from a perf perspective, but the use of () instead of [] was a cute pun, and could have gently pushed people in the lazy direction. But then, if combinators are better for the lazy case, then the case for eager traversal is less strong -- and so we end up in current situation of punting on both of them for now.
In many ways it's too bad -- Array.map is usually much worse than a
for-of and Array.push, which is what an array comprehension is, and
using combinators means you can't yield in the comprehended expression
([for (x of y) ... yield ...]). But the extensibility benefit of
combinators on Iterator.prototype will probably make deferal of
comprehensions a net win for many programmers.
On Mon, Aug 11, 2014 at 8:03 AM, Kevin Smith <zenparsing at gmail.com> wrote:
I still don't quite understand the motivation for removing comprehensions from ES6. They are a well-loved feature of Python and I have every reason to believe that they would be well-liked in Javascript.
I agree. It would be great to understand what happened here.
The original rationale was to make room for some more extensible comprehension syntax for ES7, but is that really going to happen? Is anyone currently interested in championing it?
Turns out, yes: you can do it by constructing a graph of combinators, with normal method syntax. So you can do:
var sequence = iterable.lazy().if(somePredicate)
That makes sense, but are comprehensions in any way future-hostile to this approach? My impression is that Python programmers find comprehensions more "approachable" than itertools-style programming. Although the combinator approach is obviously more extensible and general, comprehensions seem better suited to some cases and skill levels. Is there any reason that we can't have both?
On Mon, Aug 11, 2014 at 8:49 AM, Andy Wingo <wingo at igalia.com> wrote:
Turns out, yes: you can do it by constructing a graph of combinators, with normal method syntax.
Dave Herman put together a repository showing a real program, adapted from array comprehensions: dherman/sudoku/blob/master/solver.pythonic.js to using methods called .lazy(), .concat(), .map(), .filter(), and .flatMap(): dherman/sudoku/blob/master/solver.methods.js but I don't understand what's going on there. Where are these methods defined? Is it intended that .map() and .flatMap() will be defined on all iterables?
If so, how? Didn't the committee also decide not to give Iterables or Iterators a common prototype, or define these methods as part of the Iterator or Iterable interface?
Dave Herman put together a repository showing a real program, adapted from array comprehensions: dherman/sudoku/blob/master/solver.pythonic.js
I would also like to point out that (from my experience) this heavy, multiline usage of comprehensions is atypical from the Python point of view. Comprehensions tend to be short and sweet. That is their niche.
I would also like to point out that (from my experience) this heavy, multiline usage of comprehensions is atypical from the Python point of view. Comprehensions tend to be short and sweet. That is their niche.
Although, my experience is more limited than others'.
On Monday, August 11, 2014, Jason Orendorff <jason.orendorff at gmail.com> wrote:
The original rationale was to make room for some more extensible comprehension syntax for ES7, but is that really going to happen? Is anyone currently interested in championing it?
Yes, Dave Herman is the champion for this in ES7
On Mon 11 Aug 2014 16:14, Jason Orendorff <jason.orendorff at gmail.com> writes:
Where are [methods called .lazy(), .concat(), .map(), .filter(), and .flatMap()] defined? Is it intended that .map() and .flatMap() will be defined on all iterables?
I think the intention is to place them on a new Iterator.prototype. The notes record that a new Iterator.prototype is to be added to the spec, though specification of methods is deferred past ES6.
On Mon, Aug 11, 2014 at 7:20 AM, Kevin Smith <zenparsing at gmail.com> wrote:
I would also like to point out that (from my experience) this heavy, multiline usage of comprehensions is atypical from the Python point of view. Comprehensions tend to be short and sweet. That is their niche.
Although, my experience is more limited than others'.
No, that's standard. It's not Pythonic to have comprehensions that large, as it obscures rather than enlightens, particularly some of those crazier nested ones that Dave is using. If your comprehension doesn't fit in 80chars, you're probably doing something wrong, and should break it down to a real for loop.
Jason Orendorff wrote:
The original rationale was to make room for some more extensible comprehension syntax for ES7, but is that really going to happen? Is anyone currently interested in championing it?
Dave is waylaid by illness so I'll quickly supply the links to his gists showing alternatives to the version of Peter Norvig's Sudoku solver in JS that used array comprehensions and generator expressions:
(This is linked to from the meeting notes, under esdiscuss.org/notes/2014-07-30#4-7-revisit-comprehension-decision-from-last-meeting- of course.)
Quoting a bit from Dave's github:
Several different ports of Norvig's Sudoku solver to JavaScript, to elucidate differences between a few features:
- the originally planned ES6 comprehensions;
- a possible ES7 proposal to generalize and supersede those comprehensions;
- combinator methods with arrow-function shorthand
This is a cleaned up version of this JavaScript port.
Jason Orendorff wrote:
If so, how? Didn't the committee also decide not to give Iterables or Iterators a common prototype, or define these methods as part of the Iterator or Iterable interface?
The plan is to leave the iteration protocol "duck-typed" or "structural", but enable reuse of common array-extra-like methods via Iterator.prototype (which will be on the generator iterator objects' prototype chains).
The Iterator function can be used as an adaptor, to go from any old iterator (duck-typed) to an instance of the nominal type.
This isn't an attempt to force nominal-only iteration (I checked, as you would expect!). It's an "adaptor" pattern we see in lots of places in JS, which gives a prototypal home for the "itertools" methods. Methods, not functions, because JS has both but favors methods in most of its libraries starting with the stdlib.
The plan is to leave the iteration protocol "duck-typed" or "structural", but enable reuse of common array-extra-like methods via Iterator.prototype (which will be on the generator iterator objects' prototype chains).
The Iterator function can be used as an adaptor, to go from any old iterator (duck-typed) to an instance of the nominal type.
That sounds good (although I haven't thought it through), but is that plan incompatible with comprehensions as a syntactic convenience?
Tab Atkins Jr. wrote:
No, that's standard. It's not Pythonic to have comprehensions that large, as it obscures rather than enlightens, particularly some of those crazier nested ones that Dave is using.
Dave was translating mostly-mechanically from prior source.
If your comprehension doesn't fit in 80chars, you're probably doing something wrong, and should break it down to a real for loop.
See the original in Python from Peter Norvig, and an ES6-with-comprehensions version based on my JS1.8 "port" of Peter's original:
mxr.mozilla.org/mozilla-central/source/js/src/tests/ecma_6/Comprehensions/sudoku.js?raw=1
Did you find Peter's comprehensions overlong? I did not, but that's not to say anything about the original is "easy". It's a great read, especially in the posted "literate code" form.
Beyond the conciseness, comprehensions should afford engines the ability to optimize and parallelize. One of the breakthroughs in removing comprehensions from ES6 was everyone wanting to generalize from 1 (Array) to N>>1 (iterables, lazy and eager; ParallelArray; other array-likes).
Rather than adding mappar or pmap, one can use receiver-dispatched methods polymorphically as in JS funky-OOP today, no need for new delimiters after [] and () around for/of/if comprehension syntax. We really are out of delimiters!
Kevin Smith wrote:
That sounds good (although I haven't thought it through), but is that plan incompatible with comprehensions as a syntactic convenience?
Which comprehensions? Array comps, no, but see my next mail about the generalized from 1 to N point, no pmap/mappar, etc.
Generator expressions AKA generator comprehensions are less well-motivated, even though laziness is absolutely required for (e.g.) Norvig's Sudoku solver not to diverge. The use of () -- already an overloaded delimiter pair -- and the 1 vs. N problem bite harder there.
Sure, we could have array comprehensions and the more general LINQ-style syntax+protocols together. But it's plausible based on Dave's gists that we won't need much or any syntax, and TC39 isn't willing to call that right now given how ES6 has slipped. So "defer" or "cut" (what's the diff? Main thing is to get to rapid-er release and ES7/8/etc. or "annuals") won.
Thanks for the explanation. : )
On Mon, Aug 11, 2014 at 11:13 AM, Brendan Eich <brendan at mozilla.org> wrote:
Dave was translating mostly-mechanically from prior source.
Yeah, but the result was code that violated common coding guidelines from Python, and so wasn't very illuminating for the question of "what syntax should we use for comprehensions?".
Did you find Peter's comprehensions overlong? I did not, but that's not to say anything about the original is "easy". It's a great read, especially in the posted "literate code" form.
No, all of Peter's comprehensions are short and easy to comprehend. Comparing it with Dave's version, I think it's simply a matter of the JS comprehensions requiring more syntax, and crossing the line from "easy to read" to "hard to read". Here's the one bit in particular that stands out:
Norvig:
unitlist = ([cross(rows, c) for c in cols] +
[cross(r, cols) for r in rows] +
[cross(rs, cs) for rs in ('ABC','DEF','GHI') for cs in ('123','456','789')])
Herman:
var unitlist
= [for (c of cols)
cross(rows, [c])]
.concat([for (r of rows)
cross([r], cols)])
.concat([for (rs of ["ABC","DEF","GHI"])
for (cs of ["123","456","789"])
cross(rs.split(""), cs.split(""))]);
Norvig's is simple and easy to read. Herman's is polluted with additional method calls and more indentation. If we remove the indentation...
var unitlist
= [for (c of cols) cross(rows, [c])]
.concat([for (r of rows) cross([r], cols)])
.concat([for (rs of ["ABC","DEF","GHI"]) for (cs of ["123","456","789"]) cross(rs.split(""), cs.split(""))]);
Now pretend that for-of works on strings... (maybe it already does, for all I know, and Dave was just being paranoid)
var unitlist
= [for (c of cols) cross(rows, c)]
.concat([for (r of rows) cross(r, cols)])
.concat([for (rs of ["ABC","DEF","GHI"]) for (cs of ["123","456","789"]) cross(rs, cs)]);
Okay, it's still a little bit harder to read, due to the .concat() calls and their attendant additional parens, and the parens around the for-of bodies, but it's not terribly worse. It's still bad enough that I don't know if I'd write it.
(Man, now that I've looked at these for a bit, I really see the logic of Python putting the value before the for. It gives you an examplar value of what the array will look like, right up front, before sliding in the "and here's how you make the rest of them!" indicator. It also means that all three constructions have the cross() call, the important part, lined up in front so you can see the parallel structure more easily. This really helps me comprehend the code, no pun intended.)
Beyond the conciseness, comprehensions should afford engines the ability to optimize and parallelize. One of the breakthroughs in removing comprehensions from ES6 was everyone wanting to generalize from 1 (Array) to N>>1 (iterables, lazy and eager; ParallelArray; other array-likes).
Rather than adding mappar or pmap, one can use receiver-dispatched methods polymorphically as in JS funky-OOP today, no need for new delimiters after [] and () around for/of/if comprehension syntax. We really are out of delimiters!
Oh, I agree with all this. Python's out of delimiters, too, they just happened to have had one more set available, so they could do dict/set comprehensions. They're stuck now as well. ^_^
(Man, now that I've looked at these for a bit, I really see the logic of Python putting the value before the for. It gives you an examplar value of what the array will look like, right up front, before sliding in the "and here's how you make the rest of them!" indicator. It also means that all three constructions have the cross() call, the important part, lined up in front so you can see the parallel structure more easily. This really helps me comprehend the code, no pun intended.)
I agree - having the "for" in front makes it harder to read.
Tab Atkins Jr. wrote:
Okay, it's still a little bit harder to read, due to the .concat() calls and their attendant additional parens, and the parens around the for-of bodies, but it's not terribly worse. It's still bad enough that I don't know if I'd write it.
Strings are iterable in ES6, which helps. The stuff you're citing is of the sort where "it wasn't the last paren I ate that made me fat" and "every little bit helps". That's why when I first proposed for ES4 and implemented comprehensions in JS1.7-8, I used paren-free head syntax. But paren-free has issues with comp-expr at end (more on this below).
Can't help lack of + on arrays until we get operators sorted out (ES7, progress made at last f2f). So JS will suffer the .concat calls in the meanwhile.
(Man, now that I've looked at these for a bit, I really see the logic of Python putting the value before the for. It gives you an examplar value of what the array will look like, right up front, before sliding in the "and here's how you make the rest of them!" indicator. It also means that all three constructions have the cross() call, the important part, lined up in front so you can see the parallel structure more easily. This really helps me comprehend the code, no pun intended.)
Me too, but I lost that battle a while ago. The "LTR" forces prevailed and I don't see much point in reopening, except to compare unbiased Python sources such as Peter's solver to various ports. The "RTL" comprehension style is Math-y and again works best with short quantifiers and qualifiers. Comprehensions should be one-liners, with rare exceptions such as unitlist's initialiser.
Kevin Smith wrote:
I agree - having the "for" in front makes it harder to read.
Another argument for deferring!
Le 11 août 2014 à 21:30, Tab Atkins Jr. <jackalmage at gmail.com> a écrit :
Okay, it's still a little bit harder to read, due to the .concat() calls and their attendant additional parens, and the parens around the for-of bodies, but it's not terribly worse. It's still bad enough that I don't know if I'd write it.
I think I'd have written (without comprehension):
var unitlist = [...function*() {
for (let c of cols)
yield cross(rows, c)
for (let r of rows)
yield cross(r, cols)
for (let rs of ["ABC","DEF","GHI"])
for (let cs of ["123","456","789"])
yield cross(rs, cs)
}]
although it is somewhat lengthier because of extra function*, lets and yields.
Another argument for deferring!
Yep. I think I'm satisfied now.
On Mon 11 Aug 2014 21:30, "Tab Atkins Jr." <jackalmage at gmail.com> writes:
Now pretend that for-of works on strings... (maybe it already does, for all I know
It does. It iterates over code-points in the string. http://people.mozilla.org/~jorendorff/es6-draft.html#sec-%stringiteratorprototype%.next
Where are [methods called .lazy(), .concat(), .map(), .filter(), and .flatMap()] defined? Is it intended that .map() and .flatMap() will be defined on all iterables?
Just in case it wasn't obvious from the repo, these are defined in traceur.init.js. Lovely style.
I have to admit that the sudoku demos were quite an eye opener in terms of just what the features in es6 can do.
On the other hand, I agree with others that typical use of array comprehensions are simple and fast (see fast.js) due to not having function call overhead each iteration. I'm concerned that the sexy use of yield and function* may not be as performant as simple comprehensions.
-- Owen
I still don't quite understand the motivation for removing comprehensions from ES6. They are a well-loved feature of Python and I have every reason to believe that they would be well-liked in Javascript. For simplicity, we could certainly leave off array comprehensions, while keeping generator comprehensions, since you can build a list from a comprehension using spread syntax:
Can someone explain more clearly the motivation for removing this feature?
I still don't quite understand the motivation for removing comprehensions from ES6. They are a well-loved feature of Python and I have every reason to believe that they would be well-liked in Javascript. For simplicity, we could certainly leave off array comprehensions, while keeping generator comprehensions, since you can build a list from a comprehension using spread syntax: var list = [...(for x of iterable if somePredicate(x)]; Can someone explain more clearly the motivation for removing this feature? Thanks, Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.mozilla.org/pipermail/es-discuss/attachments/20140811/21d84024/attachment.html>