Standard @iter module unfriendly to collection builders

# Allen Wirfs-Brock (12 years ago)

The Iterators proposal includes the definition (harmony:iterators#standard_api ) of functions that are intended to support various common iteration patterns.

For example,

for ((k of keys(x)) { ...} for (v of values(x)) { ...} for ([k,v] of items(x)) {...} for (k of allKeys(x)) { ...} for (k of allValues(x)) { ...} for (i of allItems(x)) { ...}

The use of these functions seems to be pretty much an essential part of the intended use of the for-of statement.

Most object-oriented framework designer would immediately see a flaw in this design. keys, values, items, etc. are all functions that incorporate into them a specific collection element storage model. This makes it impossible to use them with new collections that uses a different storage model. For example, they would not work over a binary tree based collection.

In providing better forms of iteration for the language, it is important that we do not box it into a specific collection implementation. We want ES to be a first tier language that fully supports the creation of rich collection abstractions with a broad range of functional and performance trade-off.

Operations, such as these iteration functions need to be dynamically dispatched, based upon the type of collection they are applied to. A OO designer would simply avoid this problem by replacing these functions with methods (or getter properties):

for ((k of x.keys) { ...} for (v of x.values) { ...} for ([k,v] of x.items) {...} for (k of x.allKeys) { ...} for (k of x.allValues) { ...} for (i of x.allItems) { ...}

However, some may find this formulation less attractive or have other reasons to prefer a functional formulation. If we perfer to adopt the functional formulation, then it should be done in a manner that does the necessary dynamic dispatch. For example:

export function keys(obj) { return { [iterator]: function(() {return obj.keys} } }

Note that if there was a concern about property name conflicts with "keys", "values", etc. and the data storage keys of property access based collections, then built-in private names should be specified instead of these names for the corresponding method/getter property names.

There might also be a concern that this forces all collection implementors to code these iterator methods. That issue could be avoided by defining the built-in functions to conditionally check for the corresponding methods of the collection object and to fall back to a default implementation. However, it isn't obvious that the property based backing store is the best long term default for collections.


A different issue: it isn't clear why the distinction between own and inherited properties are built into these iterator functions. This is not a distinction that would apply to many new collection abstractions.

Even if it is a useful distinction,making "own property" iteration be the default seems wrong. The inheritance structure of an object should be an object implementation issue, not an object client usage issue.

For example, whether a call: let collection = (new Collection("a",1,"b",2,"c",3)).update("a",4"); results in an object that looks like {a:4,b:2,c:3} or an object that looks like {a:1,b:2,c:3}<|{a:4} should not be relevant to most clients.

We are putting lots of energy into trying to support better object-oriented abstraction capabilities, up to and including class literals. We should also be making sure that the other features we add to ES.next also support good object-oriented design principles.

# Brendan Eich (12 years ago)

On Nov 12, 2011, at 11:26 AM, Allen Wirfs-Brock wrote:

The Iterators proposal includes the definition (harmony:iterators#standard_api ) of functions that are intended to support various common iteration patterns.

For example,

for ((k of keys(x)) { ...} for (v of values(x)) { ...} for ([k,v] of items(x)) {...} for (k of allKeys(x)) { ...} for (k of allValues(x)) { ...} for (i of allItems(x)) { ...}

The use of these functions seems to be pretty much an essential part of the intended use of the for-of statement.

The prior question is what, if anything,

for (x of y) ... // and [x for x of y], etc.

means.

Should it throw if there's no @iterator private-named property in y and y is not a Proxy with an iterate trap? That is our current thinking. It may not be reflected well in the wiki.

The alternative is to iterate over property values of an object denoted y that has no unstratified @iterator or stratified handler iterate trap. But that is hostile to collections and your [] proposal.

OTOH I see no problem for collection writers if we make for-of throw on untrapped objects. Collection authors would bind @iterator, @elementGet, and @elementSet. Life would be grand. Right?

# Erik Arvidsson (12 years ago)

Another thing to consider is whether these functions belong in an iter module or in a reflect module? I'm leaning towards the letter.

# Brendan Eich (12 years ago)

On Nov 13, 2011, at 10:17 AM, Erik Arvidsson wrote:

Another thing to consider is whether these functions belong in an iter module or in a reflect module? I'm leaning towards the letter.

Does it matter, apart from the name?

I'd rather have more and more precisely defined modules than one reflect dumping ground. JS has built-in reflection, from day 1, arguably overloaded with base-level operations but part of the package deal. We won't be factoring all such unstratified reflection out.

The "@iter" name is short and Python-inspired, it should match for-of loops, comprehensions, and generators. It could grow too large, but it wouldn't be larger than "@reflect".

If we end up with a standard prelude, some of these helpers could be imported by it.

# Allen Wirfs-Brock (12 years ago)

On Nov 12, 2011, at 12:05 PM, Brendan Eich wrote:

On Nov 12, 2011, at 11:26 AM, Allen Wirfs-Brock wrote:

The Iterators proposal includes the definition (harmony:iterators#standard_api ) of functions that are intended to support various common iteration patterns.

For example,

for ((k of keys(x)) { ...} for (v of values(x)) { ...} for ([k,v] of items(x)) {...} for (k of allKeys(x)) { ...} for (k of allValues(x)) { ...} for (i of allItems(x)) { ...}

The use of these functions seems to be pretty much an essential part of the intended use of the for-of statement.

The prior question is what, if anything,

for (x of y) ... // and [x for x of y], etc.

means.

Should it throw if there's no @iterator private-named property in y and y is not a Proxy with an iterate trap? That is our current thinking. It may not be reflected well in the wiki.

Another possibility would be to build in default @iterator methods on Object.prototype and Array.prototype. I'd probably make the object implementation be an items for enumerable properties and the array implementation iterate the values of 0..length-1 .

The alternative is to iterate over property values of an object denoted y that has no unstratified @iterator or stratified handler iterate trap. But that is hostile to collections and your [] proposal.

Property values (without key identification) doesn't sounds that generally useful.

OTOH I see no problem for collection writers if we make for-of throw on untrapped objects. Collection authors would bind @iterator, @elementGet, and @elementSet. Life would be grand. Right?

Yes, suspect that people defining collections would routinely want to over-ride @iterator. However, the defaults probably do establish some expectations. Particularly if we aren't providing many (any?) built-in collections that might establish other expectations.

# Brendan Eich (12 years ago)

On Nov 14, 2011, at 9:07 AM, Allen Wirfs-Brock wrote:

On Nov 12, 2011, at 12:05 PM, Brendan Eich wrote:

On Nov 12, 2011, at 11:26 AM, Allen Wirfs-Brock wrote:

The Iterators proposal includes the definition (harmony:iterators#standard_api ) of functions that are intended to support various common iteration patterns.

For example,

for ((k of keys(x)) { ...} for (v of values(x)) { ...} for ([k,v] of items(x)) {...} for (k of allKeys(x)) { ...} for (k of allValues(x)) { ...} for (i of allItems(x)) { ...}

The use of these functions seems to be pretty much an essential part of the intended use of the for-of statement.

The prior question is what, if anything,

for (x of y) ... // and [x for x of y], etc.

means.

Should it throw if there's no @iterator private-named property in y and y is not a Proxy with an iterate trap? That is our current thinking. It may not be reflected well in the wiki.

Another possibility would be to build in default @iterator methods on Object.prototype and Array.prototype. I'd probably make the object implementation be an items for enumerable properties and the array implementation iterate the values of 0..length-1 .

Not a bad set of defaults if Object.prototype was not a prototype of Array.prototype, and I was inclined the same way once upon a time.

But then Jason (cc'ed) made a good argument (which I resisted fiercely at first because of the obvious utility of the defaults) that doing so in ES.next means anyone customizing collection iteration later will break generic code that wants only items, or only values.

Really, JS is different from Python in this way: Array and Object are more like list+object and dict+object hybrids, and in Python class instances (objects) throw without a custom iter. They do not act like lists or dicts.

Generic code will suffer, and type confusion bugs will tend to be more common ceteris paribus, if we do what you suggest.

I tried patching the defaults to be value-only iteration. That still is future-hostile to evolving collection @iterator settings, but it's better.

Yes, suspect that people defining collections would routinely want to over-ride @iterator. However, the defaults probably do establish some expectations. Particularly if we aren't providing many (any?) built-in collections that might establish other expectations.

We might need to do some work there, with the community and prototype implementations.

# Erik Arvidsson (12 years ago)

On Mon, Nov 14, 2011 at 09:07, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:

Another possibility would be to build in default @iterator methods on Object.prototype and Array.prototype.  I'd probably make the object implementation be an items for enumerable properties and the array implementation iterate the values of 0..length-1 .

Defining @iterator on Object.prototype seems like it would be a disservice to future js programmers. It would basically turn the for-of loop into another reflection based loop and we would be back to square one.

I'm in favor of Array.prototype. at iterator though

# Brendan Eich (12 years ago)

On Nov 14, 2011, at 11:59 AM, Erik Arvidsson wrote:

On Mon, Nov 14, 2011 at 09:07, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:

Another possibility would be to build in default @iterator methods on Object.prototype and Array.prototype. I'd probably make the object implementation be an items for enumerable properties and the array implementation iterate the values of 0..length-1 .

Defining @iterator on Object.prototype seems like it would be a disservice to future js programmers. It would basically turn the for-of loop into another reflection based loop and we would be back to square one.

I'm in favor of Array.prototype. at iterator though

Yes, good point! I forgot to say that in my last reply. That seems unproblematic since anything delegating to Array.prototype should be array-like in wanting for-of to iterate values in index order, skipping holes.

# Jason Orendorff (12 years ago)

On Sun, Nov 13, 2011 at 12:17 PM, Erik Arvidsson <erik.arvidsson at gmail.com> wrote:

Another thing to consider is whether these functions belong in an iter module or in a reflect module? I'm leaning towards the letter.

I completely agree.

for-of should be the tool for iterating over collections.

We should make it super easy to get a collection of an object's properties.

The right name for that facility is @reflect or @inspect, not @iter. The appropriate Python analog is the inspect module, not itertools. docs.python.org/library/inspect.html

# Jason Orendorff (12 years ago)

To address Allen's original question:

I think the Map and Set classes in Harmony mean that not all structured data is stored as object properties. I think that's a good thing.

However it does mean that we must have a separation of concerns between

  • iterating over a collection
  • object property inspection ...because I don't see how 'for (p of obj)' can be both the right syntax for walking an arbitrary object's properties and the right syntax for walking a Set's elements. Someday the "arbitrary object" will be a Set. And then what?

So if we take that separation of concerns as our maxim, what do we end up with? Here's a sketch.

Basics: for (v of arr) // each element of an Array for (v of set) // each value in a Set for (k of map) // each key in a Map, following Python

Host objects / libraries: for (elt of document.getElementsByTagName('P')) // DOM for (elt of $("div.main p")) // hypothetical jQuery support

Other styles of iteration: for ([i, v] of arr.items()) // index-value pairs (new Array method) for ([k, v] of map.items()) // key-value pairs for (v of map.values()) // just values, no keys (uncommon use case)

Enumerating properties: import * from '@inspect'; for (p of keys(obj)) for (v of values(obj)) for ([p, v] of items(obj)) Or if you don't want to import anything: for (p of Object.keys(obj)) for (p of Object.getOwnPropertyNames(obj))

This makes Map slightly nicer to iterate over than Object. I think Map is a better, less error-prone Map than Object anyway, so that's appropriate.

# Brendan Eich (12 years ago)

On Nov 15, 2011, at 1:42 PM, Jason Orendorff wrote:

On Sun, Nov 13, 2011 at 12:17 PM, Erik Arvidsson <erik.arvidsson at gmail.com> wrote:

Another thing to consider is whether these functions belong in an iter module or in a reflect module? I'm leaning towards the letter.

I completely agree.

for-of should be the tool for iterating over collections.

We should make it super easy to get a collection of an object's properties.

The right name for that facility is @reflect or @inspect, not @iter. The appropriate Python analog is the inspect module, not itertools. docs.python.org/library/inspect.html

I defer to you, oh Python master. ;-)

But still, @reflect is gonna be big. Arguably a bunch of things in JS from day 1 up through ES5.1 should go there, or appear there in better and more stylish clothes.