Discuss
  • Yet more ambiguities in property enumeration

Yet more ambiguities in property enumeration

# Mark S. Miller (15 years ago)

Mike Stay (cc'ed) noticed the following ambiguity:

What should the following print?

var base = {x:8};
function showProps(obj) {
  var result = [];
  for (var k in obj) {
    result.push(k, ': ', ''+obj[k], '\n');
  }
  return result.join('');
}
var derived = Object.create(base, {x: {value: 9, enumerable: false}});
showProps(derived);

Of current ES5 implementations in progress,

  • TraceMonkey and SES5/3 (Mike Stay's emulation of the Secure subset of EcmaScript 5 strict on current ES3R browsers) currently gives the empty string, meaning that the non-enumerable "x" in derived shadows the enumerable "x" from base.

  • WebKit nightly and V8 currently gives "x: 9" meaning that the enumerable "x" from base shows through, causing the above loop to look up the value of the unenumerable shadowing "x" property.

I have not tested any other partial ES5 implementations (Rhino, ObjectPascal, ??).

The relevant test seems to be ES5 section 12.6.4 step 6.a (and likewise 7.a for the next production):

  Let P be the name of the next property of obj whose [[Enumerable]]

attribute is true.

# Allen Wirfs-Brock (15 years ago)

While 12.6.4 is does not explicitly talk about this case I think it implicitly covers it.

Line 6.a (of the first algorithm, 7.a of the second) says "Let P be the name of the next property of obj whose [[Enumerable]] property is true". I don't believe that there is any other operation or function in ES5 that when doing a property access by-passes a own property based upon some condition and instead accesses as shadowed inherited property. Given that it's pretty clear that the quoted sentence should be read as if it said "Let P be the name of the next property accessible via the [[Get]] internal method of obj whose [[Enumerable]] property is true"

Also note that the final paragraph says "a property of a prototype is not enumerated if it is "shadowed" because some previous object in the prototype chain has a property with the same name". Note this just says "shadowed" which I believe (I probably wrote it :-) show be interpreted in the accessible via [[Get]] sense. In particular, it does not say something like 'has the same name of a enumerated property of some previous object in the prototype chain"

Given these two points of the specification, I think it takes some pretty creative interpretation to arrive at the Webkit/V8 interpretation. I suspect, that their behavior is probably an implementation artifact rather than something intentional. I can enter the more explicit formulation of (6.a/7.a) into the errata if there is a consensus that it is really needed, but I think we are also fine without it.

Allen

From: es-discuss-bounces at mozilla.org [mailto:es-discuss-bounces at mozilla.org] On Behalf Of Mark S. Miller Sent: Friday, April 23, 2010 9:01 AM To: es5-discuss at mozilla.org; es-discuss Cc: Mike Stay Subject: Yet more ambiguities in property enumeration

Mike Stay (cc'ed) noticed the following ambiguity:

What should the following print?

var base = {x:8};
function showProps(obj) {
  var result = [];
  for (var k in obj) {
    result.push(k, ': ', ''+obj[k], '\n');
  }
  return result.join('');
}
var derived = Object.create(base, {x: {value: 9, enumerable: false}});
showProps(derived);

Of current ES5 implementations in progress,

  • TraceMonkey and SES5/3 (Mike Stay's emulation of the Secure subset of EcmaScript 5 strict on current ES3R browsers) currently gives the empty string, meaning that the non-enumerable "x" in derived shadows the enumerable "x" from base.

  • WebKit nightly and V8 currently gives "x: 9" meaning that the enumerable "x" from base shows through, causing the above loop to look up the value of the unenumerable shadowing "x" property.

I have not tested any other partial ES5 implementations (Rhino, ObjectPascal, ??).

The relevant test seems to be ES5 section 12.6.4 step 6.a (and likewise 7.a for the next production):

  Let P be the name of the next property of obj whose [[Enumerable]] attribute is true.
# Oliver Hunt (15 years ago)

On May 7, 2010, at 10:22 AM, Allen Wirfs-Brock wrote:

While 12.6.4 is does not explicitly talk about this case I think it implicitly covers it.

Line 6.a (of the first algorithm, 7.a of the second) says “Let P be the name of the next property of obj whose [[Enumerable]] property is true”. I don’t believe that there is any other operation or function in ES5 that when doing a property access by-passes a own property based upon some condition and instead accesses as shadowed inherited property. Given that it’s pretty clear that the quoted sentence should be read as if it said “Let P be the name of the next property accessible via the [[Get]] internal method of obj whose [[Enumerable]] property is true”

Also note that the final paragraph says “a property of a prototype is not enumerated if it is “shadowed” because some previous object in the prototype chain has a property with the same name”. Note this just says “shadowed” which I believe (I probably wrote it :-) show be interpreted in the accessible via [[Get]] sense. In particular, it does not say something like ‘has the same name of a enumerated property of some previous object in the prototype chain”

Given these two points of the specification, I think it takes some pretty creative interpretation to arrive at the Webkit/V8 interpretation. I suspect, that their behavior is probably an implementation artifact rather than something intentional. I can enter the more explicit formulation of (6.a/7.a) into the errata if there is a consensus that it is really needed, but I think we are also fine without it.

The JSC implementation is deliberate (and i believe brendan has said that SM will be moving to match it), the interpretation is that enumeration is a breadth first search of all enumerable properties of an object that are present at the start of iteration, in the order of insertion, skipping any properties that subsequently disappear from the object.

The algorithm (as implemented in JSC and V8) for 'for (lhs in object) S' is essentially:

names = enumerablePropertyNames(object) for (var i = 0; i < names.length; i++) { if (!names[i] in object) continue; lhs = names[i]; S }

with function enumerablePropertyNames(object) { var result = []; while (object) { var keys = Object.keys(object); for (var i = 0; i < keys.length; i++) { if (result.indexOf(keys[i]) != -1) continue; result.push(keys[i]); } } return result; }

I'm 99% sure this code is correct, but i just typed it into Mail so there maybe typos

# Allen Wirfs-Brock (15 years ago)

I don't think it is valid to use Object.keys as part of your definition of for-in because that leaves you with a circular definition as the specification for Object.keys says it orders the keys in the same order used by for-in.

Ignoring algorithms for the moment and simply looking at semantics, I don't see how your interpretation makes any sense at all. The semantics of prototype inheritance creates the illusion that inherited properties are part of the inheriting object unless they are explicitly over-ridden by an own property. The semantics you describe break this illusion by essentially allowing the [[enumerable]] attribute of an inherited property to over-ride the [[enumerable]] attribute of a same-named own property. This is essentially the same mistake that the JScript has historically made with ES1-3 when it allowed an inherited dontenum attribute to suppress the enumeration of a like-name own property. I think there is universal agreement that JScript was wrong in this regard. What you are proposing is equally wrong but with the boolean value of the attribute reversed.

I believe that the correct list of enumerated names for for-in (in the absence of side-effects that modify properties or property attributes) is:

function getEnumerableNames(obj) { var encounteredKeys= Object.create(null); var result = []; while (obj) { Object.getOwnPropertyNames(obj).forEach(function(n) { if (!(n in encounteredKeys)) { encounteredKeys[n]=true; if (Object.getOwnPropertyDescriptor(obj,n).enumerable) result.push(n); } } obj=Object.getPrototypeOf(obj); } return result; }

This is also the list of names that Object.keys should produce. The actual order of the names in the result of the above function is not specified by ES5. The prose of 12.6.4 (but not the algorithm) concerning deleted properties means that side-effects that delete properties should be handled as if for-in was then implemented as:

getEnumerableNames(obj).forEach(function(n) {if (n in obj) execute-for-in-body-with-n});

But note that since the order is undefined, if the body does any deletes there is no guarantee whether you will or won't see a key that is subject to deletion. The only guarantee is that if all occurrences of a particular property name (there may be multiple along the prototype chain) are deleted before processing that name, then the for-in body will not be execute for that name.

Things are even more unspecified if properties are added as side-effects or if enumerable attribute values are dynamically changed.

However, I believe the most important use case is the one where there are no evil side-effects in the for-in body. For that case the set of property names for which the for-in body is executed is supposed to be the set generated by the above getEnumerableNames function.

Allen

From: Oliver Hunt [mailto:oliver at apple.com] Sent: Friday, May 07, 2010 10:40 AM To: Allen Wirfs-Brock Cc: Mark S. Miller; es5-discuss at mozilla.org; es-discuss; Mike Stay Subject: Re: Yet more ambiguities in property enumeration

On May 7, 2010, at 10:22 AM, Allen Wirfs-Brock wrote:

While 12.6.4 is does not explicitly talk about this case I think it implicitly covers it.

Line 6.a (of the first algorithm, 7.a of the second) says "Let P be the name of the next property of obj whose [[Enumerable]] property is true". I don't believe that there is any other operation or function in ES5 that when doing a property access by-passes a own property based upon some condition and instead accesses as shadowed inherited property. Given that it's pretty clear that the quoted sentence should be read as if it said "Let P be the name of the next property accessible via the [[Get]] internal method of obj whose [[Enumerable]] property is true"

Also note that the final paragraph says "a property of a prototype is not enumerated if it is "shadowed" because some previous object in the prototype chain has a property with the same name". Note this just says "shadowed" which I believe (I probably wrote it :-) show be interpreted in the accessible via [[Get]] sense. In particular, it does not say something like 'has the same name of a enumerated property of some previous object in the prototype chain"

Given these two points of the specification, I think it takes some pretty creative interpretation to arrive at the Webkit/V8 interpretation. I suspect, that their behavior is probably an implementation artifact rather than something intentional. I can enter the more explicit formulation of (6.a/7.a) into the errata if there is a consensus that it is really needed, but I think we are also fine without it.

The JSC implementation is deliberate (and i believe brendan has said that SM will be moving to match it), the interpretation is that enumeration is a breadth first search of all enumerable properties of an object that are present at the start of iteration, in the order of insertion, skipping any properties that subsequently disappear from the object.

The algorithm (as implemented in JSC and V8) for 'for (lhs in object) S' is essentially:

names = enumerablePropertyNames(object) for (var i = 0; i < names.length; i++) { if (!names[i] in object) continue; lhs = names[i]; S }

with function enumerablePropertyNames(object) { var result = []; while (object) { var keys = Object.keys(object); for (var i = 0; i < keys.length; i++) { if (result.indexOf(keys[i]) != -1) continue; result.push(keys[i]); } } return result; }

I'm 99% sure this code is correct, but i just typed it into Mail so there maybe typos

--Oliver

Allen

From: es-discuss-bounces at mozilla.org<mailto:es-discuss-bounces at mozilla.org> [mailto:es-discuss-bounces at mozilla.org] On Behalf Of Mark S. Miller

Sent: Friday, April 23, 2010 9:01 AM To: es5-discuss at mozilla.org<mailto:es5-discuss at mozilla.org>; es-discuss

Cc: Mike Stay Subject: Yet more ambiguities in property enumeration

Mike Stay (cc'ed) noticed the following ambiguity:

What should the following print?

var base = {x:8};
function showProps(obj) {
  var result = [];
  for (var k in obj) {
    result.push(k, ': ', ''+obj[k], '\n');
  }
  return result.join('');
}
var derived = Object.create(base, {x: {value: 9, enumerable: false}});
showProps(derived);

Of current ES5 implementations in progress,

  • TraceMonkey and SES5/3 (Mike Stay's emulation of the Secure subset of EcmaScript 5 strict on current ES3R browsers) currently gives the empty string, meaning that the non-enumerable "x" in derived shadows the enumerable "x" from base.

  • WebKit nightly and V8 currently gives "x: 9" meaning that the enumerable "x" from base shows through, causing the above loop to look up the value of the unenumerable shadowing "x" property.

I have not tested any other partial ES5 implementations (Rhino, ObjectPascal, ??).

The relevant test seems to be ES5 section 12.6.4 step 6.a (and likewise 7.a for the next production):

  Let P be the name of the next property of obj whose [[Enumerable]] attribute is true.
# Oliver Hunt (15 years ago)

On May 7, 2010, at 12:15 PM, Allen Wirfs-Brock wrote:

I don’t think it is valid to use Object.keys as part of your definition of for-in because that leaves you with a circular definition as the specification for Object.keys says it orders the keys in the same order used by for-in.

I wasn't meaning to use this as a spec worthy definition, but as a way to see how for in was evaluated across object and it's prototypes.

Ignoring algorithms for the moment and simply looking at semantics, I don’t see how your interpretation makes any sense at all. The semantics of prototype inheritance creates the illusion that inherited properties are part of the inheriting object unless they are explicitly over-ridden by an own property. The semantics you describe break this illusion by essentially allowing the [[enumerable]] attribute of an inherited property to over-ride the [[enumerable]] attribute of a same-named own property. This is essentially the same mistake that the JScript has historically made with ES1-3 when it allowed an inherited dontenum attribute to suppress the enumeration of a like-name own property. I think there is universal agreement that JScript was wrong in this regard. What you are proposing is equally wrong but with the boolean value of the attribute reversed.

This isn't a proposal, this is what we do, and we do it this way because in the past our failure to do so broke sites.

I believe that the correct list of enumerated names for for-in (in the absence of side-effects that modify properties or property attributes) is: ...<snip>... This is also the list of names that Object.keys should produce. The actual order of the names in the result of the above function is not specified by ES5. The prose of 12.6.4 (but not the algorithm) concerning deleted properties means that side-effects that delete properties should be handled as if for-in was then implemented as:

getEnumerableNames(obj).forEach(function(n) {if (n in obj) execute-for-in-body-with-n});

object.keys only returns own properties -- it doesn't return the properties of the prototype.

But note that since the order is undefined, if the body does any deletes there is no guarantee whether you will or won’t see a key that is subject to deletion. The only guarantee is that if all occurrences of a particular property name (there may be multiple along the prototype chain) are deleted before processing that name, then the for-in body will not be execute for that name. Things are even more unspecified if properties are added as side-effects or if enumerable attribute values are dynamically changed.

It seems a bug in the spec that this isn't defined -- the order of enumeration of own properties in actual implementations is the order of addition, it has to be this or sites will break. JSC originally followed the ES3 spec (which didn't defined ordering) and sites broke. This is the set of constraints the I know are necessary:

  1. Order of enumeration of own properties is the order of addition
  2. You can't enter the loop with a property that has been added since the start of iteration
  3. You can't enter the loop with a property has been removed since that start of iteration
  4. You can't enter the loop with the same value multiple times

Sites break if you don't enforce any one of these constraints. We know this because the behaviour of iteration in JSC has been specifically implemented to enforce these constraints as a result of re-world site breakage.

It should be easy to see how each of these constraints is enforced in the example code i listed earlier.

The place where there has historically been a difference between implementations is precisely what happens with properties from the prototype chain.

# Oliver Hunt (15 years ago)

On May 7, 2010, at 12:56 PM, stay wrote:

Really? How could sites possibly depend on being able to mark properties non-enumerable but still have them appear in a for-in loop?

Sorry, shadowing with a non-enumerable property is new to ES5, I was meaning the behaviour i described is what we need to do to ensure sites work.

Now days my understanding is the only place where there is confusion over behaviour is purely which prototype properties should be enumerated.

# Allen Wirfs-Brock (15 years ago)

For what it's worth, here are the for-in scenarios where I believe a user can reasonably expect to get identical enumerations from all major browsers including IE8. The parenthesized items are extrapolations to ES5 features where I would also expect to not impact the results.

  1. For objects that do not inherit any enumerable properties, do not have any numeric indexed properties, and have no own properties with names that are the same name as a built-in non-enumerable property of Object.prototype, the enumeration order is the order in which the properties were added to the object. For objects created via object literals, object are conceptually added in the order that they occur within the literal. (For properties added via Object.create or Object.defineProperties the conceptual addition order is the enumeration order of the properties of the properties descriptor argument to those functions.) a. {z:1, y:2,x:3} //for-in enumeration order: z,y,x b. var obj={}; obj.a=1; obj.c=2; obj.b=3; //for-in enumeration order: a,c,b c. var obj3={z:1, y:2,x:3}; obj3.a=4; obj3.c=5; //for-in enumeration order: z,y,x,a,c d. (Object.defineProperties({z:1, y:2,x:3}, {q:{enumerable:true},r:{enumerable:true}}); //for-in enumeration order: z,y,x,q,r)

  2. For objects that do not inherit any enumerable properties and whose enumerable own properties are numeric indexed properties that are consecutive integers starting with 0 and all properties were created in a single operation in ascending numeric order, the enumeration order is ascending numeric order. a. [0,1,2,3] //enumeration order 0,1,2,3 b. {0:0, 1:1, 2:2} // enumeration order 0,1,2 c. new Array(0,1,2,3) //enumeration order 0,1,2,3 d. (new String("abc") //enumeration order 0,1,2) e. function(a,b,c){for (var p in arguments);} //enumeration order 0,1,2

In addition, for all major browsers except IE8 and earlier these additional case is also common

  1. For objects that do not inherit any enumerable properties and which do not have any numeric indexed properties, deleting a property and then adding it back to the same object moves that property to the end of the enumeration order. a. var obj1={z:1, y:2,x:3}; delete obj1.y; obj1.y=3.2; //for-in enumeration order: z,x,y

  2. For objects that have inherited enumerable properties but which do not have any numeric indexed properties (either own or inherited) or any over-ridden inherited properties the enumerations order is the enumeration of the own properties of the object, followed by the enumeration of the own properties of each object on the [[prototype]] chain in [[prototype chain order]]. The own properties of each prototype object are enumerated using rules 1 and 2 above. However, if an enumerable property exists at any level of the prototype chain that has the same name as a property that already been enumerated at a lower level, that property is not included in the enumeration. a. var obj3={z:1,x:2}; var obj2=Object.create(obj3);obj2.q=1,obj2.r=2; var obj1= Object.create(obj2);obj1.a=1,obj1.b=2;// enumeration order a,b,q,r,z,x

In all other cases there is some variation of order or included properties among major browsers.

# Allen Wirfs-Brock (15 years ago)

object.keys only returns own properties -- it doesn't return the properties of the prototype.

You're right, for some reason I have it firmly planted in my head that Object.keys is suppose to produce the same set of property names that for-in produces but that certainly isn't what the spec.

allen From: Oliver Hunt [mailto:oliver at apple.com] Sent: Friday, May 07, 2010 12:42 PM To: Allen Wirfs-Brock Cc: Mark S. Miller; es5-discuss at mozilla.org; es-discuss; Mike Stay; brendan at mozilla.com Subject: Re: Yet more ambiguities in property enumeration

On May 7, 2010, at 12:15 PM, Allen Wirfs-Brock wrote:

I don't think it is valid to use Object.keys as part of your definition of for-in because that leaves you with a circular definition as the specification for Object.keys says it orders the keys in the same order used by for-in.

I wasn't meaning to use this as a spec worthy definition, but as a way to see how for in was evaluated across object and it's prototypes.

Ignoring algorithms for the moment and simply looking at semantics, I don't see how your interpretation makes any sense at all. The semantics of prototype inheritance creates the illusion that inherited properties are part of the inheriting object unless they are explicitly over-ridden by an own property. The semantics you describe break this illusion by essentially allowing the [[enumerable]] attribute of an inherited property to over-ride the [[enumerable]] attribute of a same-named own property. This is essentially the same mistake that the JScript has historically made with ES1-3 when it allowed an inherited dontenum attribute to suppress the enumeration of a like-name own property. I think there is universal agreement that JScript was wrong in this regard. What you are proposing is equally wrong but with the boolean value of the attribute reversed.

This isn't a proposal, this is what we do, and we do it this way because in the past our failure to do so broke sites.

I believe that the correct list of enumerated names for for-in (in the absence of side-effects that modify properties or property attributes) is: ...<snip>...

This is also the list of names that Object.keys should produce. The actual order of the names in the result of the above function is not specified by ES5. The prose of 12.6.4 (but not the algorithm) concerning deleted properties means that side-effects that delete properties should be handled as if for-in was then implemented as:

getEnumerableNames(obj).forEach(function(n) {if (n in obj) execute-for-in-body-with-n});

object.keys only returns own properties -- it doesn't return the properties of the prototype.

But note that since the order is undefined, if the body does any deletes there is no guarantee whether you will or won't see a key that is subject to deletion. The only guarantee is that if all occurrences of a particular property name (there may be multiple along the prototype chain) are deleted before processing that name, then the for-in body will not be execute for that name. Things are even more unspecified if properties are added as side-effects or if enumerable attribute values are dynamically changed.

It seems a bug in the spec that this isn't defined -- the order of enumeration of own properties in actual implementations is the order of addition, it has to be this or sites will break. JSC originally followed the ES3 spec (which didn't defined ordering) and sites broke. This is the set of constraints the I know are necessary:

  1. Order of enumeration of own properties is the order of addition
  2. You can't enter the loop with a property that has been added since the start of iteration
  3. You can't enter the loop with a property has been removed since that start of iteration
  4. You can't enter the loop with the same value multiple times

Sites break if you don't enforce any one of these constraints. We know this because the behaviour of iteration in JSC has been specifically implemented to enforce these constraints as a result of re-world site breakage.

It should be easy to see how each of these constraints is enforced in the example code i listed earlier.

The place where there has historically been a difference between implementations is precisely what happens with properties from the prototype chain.

# Mark S. Miller (15 years ago)

On Thu, Jun 3, 2010 at 11:54 AM, stay <stay at google.com> wrote:

On Fri, May 7, 2010 at 1:00 PM, Oliver Hunt <oliver at apple.com> wrote:

On May 7, 2010, at 12:56 PM, stay wrote:

Really? How could sites possibly depend on being able to mark properties non-enumerable but still have them appear in a for-in loop?

Sorry, shadowing with a non-enumerable property is new to ES5, I was meaning the behaviour i described is what we need to do to ensure sites work.

Now days my understanding is the only place where there is confusion over behaviour is purely which prototype properties should be enumerated.

--Oliver

I don't think I ever saw a conclusive answer to the original question:

When a.x is enumerable, b inherits from a, and b.x is marked non-enumerable, does x get enumerated in a for-in loop or not?

If not, V8 and WebKit are doing it wrong. If so, it seems to violate the principle that properties on children shadow their parents.

Which is it?

Allen's follow-up messages talked about order of enumeration, which I'm not particularly concerned about right now.

I thought Allen's answer at < mail.mozilla.org/pipermail/es5-discuss/2010-May/003536.html> was

clear. The property is shadowed and the current WebKit & v8 behaviors are non-conformant. I have filed bug reports against them at < bugs.webkit.org/show_bug.cgi?id=38970> and <

code.google.com/p/v8/issues/detail?id=705>.

# Allen Wirfs-Brock (15 years ago)

It should not be enumerated.

In general properties are seen via the accessing object. That's what [[GetOwnProperty]] (8.12.1), which is used though the spec, does. It starts with "b" looking for 'x' walking the [[Prototype]] chain, as necessary. It stops when it finds a own property definition for 'x' and the uses the attributes of that property definition. It never looks further up the [[Prototype]] chain once such a property is found.

There are all sorts of ways that a property definition can shadow an inheritable property definition, it can change data property to an accessor property, it can make non-configurable property configurable, etc. None of these are special cased, the "closest" property definition always wins.

Remember that historically, IE behaved as follows:

var obj = {toString: false, xxx: true};
var props=[];
for (var p in obj) props.push(p);
alert(props);   //legacy IE displays "xxx" not "toString, xxx"

There was universal agreement that IE was wrong and all other browsers were correct in enumerating toString even though it shadows a non-enumerable property.

What you are describing is just the opposite situation. Consistency requires use the closest property definition and ignore what it shadows rule.

# Allen Wirfs-Brock (15 years ago)

There seems to be enough controversy about this, that it is reasonable to clarify it a bit in the errata. There are two points that I think should be covered:

First a normative statement at the end of the paragraph that begins "The mechanics and order...": A property name must not be visited more than once in any enumeration.

And a normative statement at the end of the last paragraph: The values of [[Enumerable]] attributes are not considered when determining if a property of a prototype object is shadowed by a previous object on the prototype chain.

Thoughts?

# Mark S. Miller (15 years ago)

On Tue, Jun 22, 2010 at 10:15 AM, Allen Wirfs-Brock < Allen.Wirfs-Brock at microsoft.com> wrote:

There seems to be enough controversy about this, that it is reasonable to clarify it a bit in the errata. There are two points that I think should be covered:

I haven't observed controversy so much as confusion, but perhaps I missed something. In any case, +1 on both suggestions below.