Discuss
  • "for-in", shadowing and deleting properties.

"for-in", shadowing and deleting properties.

# Gareth Smith (12 years ago)

I am trying to understand the specification of for-in, both for ES5 and for ES6, and in particular the interaction between shadowing and adding or deleting properties. Here are the parts of the ES5 spec and ES6 draft that seem most relevant:

Here are two motivating examples. In both examples, I'll assume that y is enumerated first. For each example, the question is: does the specification require x to be enumerated, or can an implementation choose to skip it?

Example 1:

var b = { x : 2 }
var a = { y : 0, __proto__ : b }

for (i in a){
    console.log(i + ": " + a[i]);
    a.x = 1
}

In this first example b.x becomes shadowed by a.x before being enumerated. Thus we have both "If new properties are added to the object being enumerated during enumeration, the newly added properties are not guaranteed to be visited in the active enumeration" and "a property of a prototype is not enumerated if it is ?shadowed? because some previous object in the prototype chain has a property with the same name."

Does the specification allow us to skip x?

Example 2:

var b = { x : 2 }
var a = { y : 0, x : 1, __proto__ : b }

for (i in a){
    console.log(i + ": " + a[i]);
    delete(a.x)
}

In this second example, a.x is deleted before it is visited. Thus, following "If a property that has not yet been visited during enumeration is deleted, then it will not be visited", it should not be visited. The b.x property is no longer shadowed, but it was not supposed to be enumerated in the first place. So can it be skipped?

An alternative reading of the specification is that we are interested in property names, not properties. Given this reading, in both examples a property named x was always reachable from a, so it must be enumerated in both examples. Is this second reading closer to the intent of the specification? I note that the current wording of the spec talks about "properties" when things are being added and deleted, and "property names" only when talking about duplicate enumerations.

# Andrea Giammarchi (12 years ago)

in the Example 1 you simply loop over all keys in the chain, included the x inherited from b.

The fact you shadow that is irrelevant, it was there in any case. The fact you get 1 in the log is because you access the shadowed property through a[i] so expected result.

You are doing a for/in, not a for/of

In the second Example is the same, for/in loops over all enumerable properties in the chain, it does not matter what you do with the current instance if that property name is inherited so, once again, expected.

var b = { x : 2 },
    a = { y : 0, __proto__ : b },
    k;

for (k in a){
    alert(k + ": " + a[k]);
    delete b.x; // note: it's b, not a
}

this will loop only over a.y skipping x

If you want to be sure about having a keys snapshot of the own properties, you can, in ES6:

let b = {x: 2},
    a = {y: 0, __proto__: b},
    k;

for (k of Object.keys(a)) {
  console.log(k);
}

or an equivalent for loop/iteration over keys such

Object.keys(a).forEach(function(k){
  this[k]; // whatever
}, a);
# Gareth Smith (12 years ago)

Andrea: Thanks for your quick reply!

I think I understand what you just told me, but would like to clarify to be sure: Am I right in thinking that you believe that to correctly interpret the for-in/Ennumerate section of the spec/draft (ES5:12.6.4 , ES6:8.3.12) we should treat the phrases "property" and "property name" as interchangeable in those sections?

I certainly think I agree with the good sense of making an implementation behave the way you have described, but am particularly interested in exactly what is and isn't expressly forbidden by the published standard.

I think you're telling me that if I write an implementation which chooses to skip the enumeration of x in either of my original examples, I will be in violation of the published standard. What I'm trying to understand now is how I should read the letter of the standards documents in order to arrive at this conclusion. Am I right in thinking that the phrases "property" and "property name" should be treated interchangeably in these sections, or is there some other phrasing or implied intent that I've misunderstood?

Should I submit versions of my original two examples as test262 cases, which check that x definitely is enumerated?

I apologise for the unusual slant of my questions. I'm not writing an ECMAScript implementation (not one intended for deployment anyway), nor am I asking exclusively about the behaviour of existing implementations. I'm part of an academic effort to formalise the existing specification in the Coq proof assistant, so I am forced to care about the detail of which particular implementation behaviours are allowed or disallowed by which particular phrases in the published standard. If we can't pin these behaviours on particular phrases in the published standard, then I'm very interested in existing community consensus.

# Andreas Rossberg (12 years ago)

mostly for historic reasons, for-in is one of the darkest corners of the ES spec. It is intentionally vague, because existing implementations differ widely in their handling of this construct.

To address your question: the spec simply does not prescribe anything coherent for this case. Have you tried different implementations? I assume they will differ, and consequently, there likely is little chance to tighten the spec for this case, nor to agree on a suitable test case for test262. It's long been acknowledged that for-in is beyond repair, which is why ES6 will have for-of as a more well-behaved replacement.

Nevertheless, it's probably worth filing a bug against the spec (bugs.ecmascript.org/describecomponents.cgi?product=ECMA-262). It could at least make clear that nothing is clear for this case. :) And yes, I can see how that sucks for JSCert.

As for your other question, yeah, I think that "property" and "property with name" are used interchangeably in those parts of the spec.

# Gareth Smith (12 years ago)

Thanks for your reply - confirmation that this is known to be a dark corner is extremely helpful :)

We have begun the process of testing the behaviour of different browsers for for-in, but still have plenty more corners to explore.

We have a possible permissive formalisation of for-in which allows all the possible behaviours I described in my original email (implementations may choose to skip the x). It sounds to me like you might think this permissive formalisation would better reflect the community consensus (that this dark corner is beyond repair) than a stricter one (in which x was guaranteed to be visited, for example).

In general, I think the message I'm getting from you is that when it comes to for-in, a more permissive reading of the standard is likely to be more accurate.

Does this seem fair?

# Andreas Rossberg (12 years ago)

On 9 May 2013 14:58, Gareth Smith <gds at doc.ic.ac.uk> wrote:

In general, I think the message I'm getting from you is that when it comes to for-in, a more permissive reading of the standard is likely to be more accurate.

Does this seem fair?

Yes, I would say so. Moreover, I wouldn't even assume that for-in semantics is deterministic for any given VM -- it can change depending on dynamic optimisations and representation changes.

# Gareth Smith (12 years ago)

Andreas Rossberg <rossberg at google.com> writes:

On 9 May 2013 14:58, Gareth Smith <gds at doc.ic.ac.uk> wrote:

In general, I think the message I'm getting from you is that when it comes to for-in, a more permissive reading of the standard is likely to be more accurate.

Does this seem fair?

Yes, I would say so. Moreover, I wouldn't even assume that for-in semantics is deterministic for any given VM -- it can change depending on dynamic optimisations and representation changes.

Thanks, that's great.

Our current candidate formalism does indeed allow for non-determinism :)

# Allen Wirfs-Brock (12 years ago)

On May 8, 2013, at 11:16 AM, Gareth Smith wrote:

...

An alternative reading of the specification is that we are interested in property /names/, not properties. Given this reading, in both examples a property named 'x' was always reachable from a, so it must be enumerated in both examples. Is this second reading closer to the intent of the specification? I note that the current wording of the spec talks about "properties" when things are being added and deleted, and "property names" only when talking about duplicate enumerations.

On May 9, 2013, at 4:58 AM, Andreas Rossberg wrote:

...

As for your other question, yeah, I think that "property" and "property with name" are used interchangeably in those parts of the spec.

My re-reading of of the ES5 language reconfirms that the distinct use of property and property name is very deliberate. Let me know if any particular phrases seem ambiguous to you.
Where I think the language is somewhat sloppy is in making a distinction being the examination of a property by the algorithm and exposing the name of a property to user code. "Visiting" might mean either of these depending upon the surrounding context.

One requirement that we agreed upon in the ES5 days was that a particular for-in loop should process any property name at most once. This was agreed upon even though we knew that some existing implementations violated that requirement in some scenarios. The language "A property name must not be visited more than once in any enumeration." was added as the statement of that requirement. the use of "property name" was very deliberate in that context.

We also agreed that if a non-enumerable property of a particular name is encountered, then that name will not be "visited" even if it is encountered for an enumerable property higher on the prototype change. My interpretation of that requirement is that it even extends to the case where the non-enumerable property is subsequently deleted and then a enumerable property of the same name is added to a prototype that has not yet been examined. Once a name has been determined to be non-enumerable, that determination can not change during an enumeration.

I've created an ES6 generator that I believe conforms to the intent of the ES5 spec: gist.github.com/allenwb/5548164 It is also permitted (not not required) to use an algorithm that was more aggressive about finding and yielding new property names that are added during the lifetime of the generator.

# Gareth Smith (12 years ago)

Hi Allen,

Allen Wirfs-Brock <allen at wirfs-brock.com> writes:

My re-reading of of the ES5 language reconfirms that the distinct use of property and property name is very deliberate. Let me know if any particular phrases seem ambiguous to you.

Ah excellent - thank you.

Where I think the language is somewhat sloppy is in making a distinction being the examination of a property by the algorithm and exposing the name of a property to user code. "Visiting" might mean either of these depending upon the surrounding context.

Ah - I think I may have been assuming that "visiting" meant exposing the name of a property to user code.

One requirement that we agreed upon in the ES5 days was that a particular for-in loop should process any property name at most once. This was agreed upon even though we knew that some existing implementations violated that requirement in some scenarios. The language "A property name must not be visited more than once in any enumeration." was added as the statement of that requirement. the use of "property name" was very deliberate in that context.

Excellent - the formalism I'm currently working on does guarantee this property.

We also agreed that if a non-enumerable property of a particular name is encountered, then that name will not be "visited" even if it is encountered for an enumerable property higher on the prototype change.

The formalism I'm currently working on seeks to guarantee the following interpretation of this requirement, which I will give a number so I can refer to it later.

(1) For some for-loop "for k in stmt", at the point which you enter a particular iteration of your loop, the property stmt[k] will be enumerable.

You are of course free to use Object.defineProperty to make it non-enumerable as soon as you enter the loop body, so it cannot be guaranteed to be enumerable at any other point in the loop body.

My interpretation of that requirement is that it even extends to the case where the non-enumerable property is subsequently deleted and then a enumerable property of the same name is added to a prototype that has not yet been examined.

Then the formalism I'm working on seems to be strictly more permissive than the one you're thinking of... Modulo my next question...

Once a name has been determined to be non-enumerable, that determination can not change during an enumeration.

Once a name has been determined to be enumerable, can that determination change to being non-enumerable? I think this ability seems to be a necessary in order to guarantee property (1) above.

I've created an ES6 generator that I believe conforms to the intent of the ES5 spec: gist.github.com/allenwb/5548164 It is also permitted (not not required) to use an algorithm that was more aggressive about finding and yielding new property names that are added during the lifetime of the generator.

Thanks!

I may well be mistaken, but I think it is possible to force your generator to appear to visit a non-enumerable property. The first way I thought of is definitely a bit convoluted:

grandproto_ob = {x:1};
proto_ob = {y:2, __proto__:grandproto_ob};
ob = {__proto__:proto_ob};
for (k in ob) {
    alert("ob."+k+" has value: "ob[k]);

    if(k===y) {
        Object.defineProperty(ob, "x", {value: 3, enumerable: false})
    }
}
// I think you'll see the following alerts:
// "ob.y has value 2"
// "ob.x has value 3"

Assuming that I've read your generator code right, and my example interacts with it in the way that I expect: then is this behaviour you intended?

If this behaviour is what you intended, then it seems to mean that my property (1) isn't weak enough.

You haven't mentioned guarantees of enumeration as (whether x is definitely guaranteed to be printed out in the two examples in my original email) as properties that everyone in ES5 agreed upon, so I'm going to continue to assume that I should specify those as permissively as possible. I'd be very interested to hear if (1) is too strong though!

# Allen Wirfs-Brock (12 years ago)

On May 9, 2013, at 9:48 AM, Gareth Smith wrote:

Hi Allen,

Allen Wirfs-Brock <allen at wirfs-brock.com> writes:

...

I may well be mistaken, but I think it is possible to force your generator to appear to visit a non-enumerable property. The first way I thought of is definitely a bit convoluted:

--8<---------------cut here---------------start------------->8--- grandproto_ob = {x:1}; proto_ob = {y:2, proto:grandproto_ob}; ob = {proto:proto_ob}; for (k in ob) { alert("ob."+k+" has value: "ob[k]);

if(k===y) { Object.defineProperty(ob, "x", {value: 3, enumerable: false}) } } // I think you'll see the following alerts: // "ob.y has value 2" // "ob.x has value 3" --8<---------------cut here---------------end--------------->8---

Assuming that I've read your generator code right, and my example interacts with it in the way that I expect: then is this behaviour you intended?

If this behaviour is what you intended, then it seems to mean that my property (1) isn't weak enough.

I think what you are doing here falls under the "newly added properties are not guaranteed to be visited" rule. However, I think this is also one of the situations where it makes a difference on whether "visited" is interpreted to mean returned to the user or processed by this algorithm. This is a case where "processed" was the intended meaning.

Behind all this was the basic idea that an implementation was not required to keep looking for new properties (or changes of attributes) in objects that have already been processed by the algorithm. With my generator algorithm and you test case, "x" only gets added to obj after the algorithm has already processed obj and moved on to proto_ob so the algorithm is allowed to ignore that added property and any shadowing effect it might of had.

You haven't mentioned guarantees of enumeration as (whether x is definitely guaranteed to be printed out in the two examples in my original email) as properties that everyone in ES5 agreed upon, so I'm going to continue to assume that I should specify those as permissively as possible. I'd be very interested to hear if (1) is too strong though!

right, (1) would require that the second predicate clause in line 10 of my generator: if (desc && desc.enumerable) yield name; be augmented with an additional search starting with the original obj for a shadowing non-numerable property with the same name. This was the sort of processing that some (most?) existing implementations didn't do and we didn't want to require it but neither did we have any agreemnt to forbid it. So, either result is acceptable. If you code is dependent upon either specific result for this case then you're screwed. Don't do it.

Allen if (desc && desc.enumerable) yield name; if (desc && desc.enumerable) yield name; if (desc && desc.enumerable) yield name;

  if (desc && desc.enumerable) yield name;
# Gareth Smith (12 years ago)

Allen Wirfs-Brock <allen at wirfs-brock.com> writes:

I think what you are doing here falls under the "newly added properties are not guaranteed to be visited" rule. However, I think this is also one of the situations where it makes a difference on whether "visited" is interpreted to mean returned to the user or processed by this algorithm. This is a case where "processed" was the intended meaning.

Ah I see - I had thought it meant "returned to the user", and therein lay my confusion.

Behind all this was the basic idea that an implementation was not required to keep looking for new properties (or changes of attributes) in objects that have already been processed by the algorithm. With my generator algorithm and you test case, "x" only gets added to obj after the algorithm has already processed obj and moved on to proto_ob so the algorithm is allowed to ignore that added property and any shadowing effect it might of had.

nods.

You haven't mentioned guarantees of enumeration as (whether x is definitely guaranteed to be printed out in the two examples in my original email) as properties that everyone in ES5 agreed upon, so I'm going to continue to assume that I should specify those as permissively as possible. I'd be very interested to hear if (1) is too strong though!

right, (1) would require that the second predicate clause in line 10 of my generator: if (desc && desc.enumerable) yield name; be augmented with an additional search starting with the original obj for a shadowing non-numerable property with the same name. This was the sort of processing that some (most?) existing implementations didn't do and we didn't want to require it but neither did we have any agreemnt to forbid it. So, either result is acceptable. If you code is dependent upon either specific result for this case then you're screwed. Don't do it.

That's really helpful, thanks!

I'll have a bit of a think about how best to update my property, then might reply again to ask if what I come up with is an accurate reflection of what you've told me.

# Gareth Smith (12 years ago)

Allen Wirfs-Brock <allen at wirfs-brock.com> writes:

I think what you are doing here falls under the "newly added properties are not guaranteed to be visited" rule. However, I think this is also one of the situations where it makes a difference on whether "visited" is interpreted to mean returned to the user or processed by this algorithm. This is a case where "processed" was the intended meaning.

Behind all this was the basic idea that an implementation was not required to keep looking for new properties (or changes of attributes) in objects that have already been processed by the algorithm. With my generator algorithm and you test case, "x" only gets added to obj after the algorithm has already processed obj and moved on to proto_ob so the algorithm is allowed to ignore that added property and any shadowing effect it might of had.

Ok, cool. I think it's clear that property(1) from my previous email was too strong. So concerning the step 6a or 7a (depending on what's on the left of the "in") in the ES5 standard, which begins "Let P be the name of the next property of obj whose [[Enumerable]] attribute is true.":

...I'm now thinking that this line might mean that we do have the following guarantee:

(2) when we enter a particular iteration of the loop for(k in obj) it must be the case that there is some obj2 in the prototype chain of obj (so obj2===obj or obj2===obj.proto and so on) such that obj2[k] is enumerable.

Does that sound right to you?

Here's one possible implementation that would violate this property(2):

--8<---------------cut here---------------start------------->8---

function *forin(obj) { let processed = new Set(); let safe_to_visit = new Array(); let safe_count = 0; while (obj!==null) { let here = Object.getOwnPropertyNames(obj); for (let i=0; i<here.length; i++) { let name = here[i]; if (processed.has(name)) continue; processed.add(name); let desc = Object.getOwnPropertyDescriptor(obj,name); if (desc && desc.enumerable) safe_to_visit[safe_count++] = name; } obj = Object.getPrototypeOf(obj); } for(let i=0; i<safe_count ; i++) { yield safe_to_visit[i]; } } --8<---------------cut here---------------end--------------->8---

However, this implementation might fall foul of the letter of the (ES5) standard, since the standard seems to require that the enumerability of the property be checked immediately before executing the loop body. What do you think?

Another possible variation on this property is:

(3) when we enter a particular iteration of the loop for(k in obj) it must be the case that at some moment since the loop started running obj[k] was enumerable.

I think your implementation at gist.github.com/allenwb/5548164 violates this property(3), since I can do:

--8<---------------cut here---------------start------------->8---

grandproto_ob = {}; Object.defineProperty(grandproto_ob, "x", {value: 1, enumerable: false}) proto_ob = {y:2, proto:grandproto_ob}; ob = {proto:proto_ob}; for (k in ob) { alert("ob."+k+" has value: "ob[k]);

if(k===y) {
    Object.defineProperty(ob, "x", {value: 3, enumerable: false})
    grandproto_ob.x = 4
}

} --8<---------------cut here---------------end--------------->8---

I think when we run this example program, there is no point at which ob.x is enumerable. I also think that when we run it using your generator, we'll see "ob.x has value: 3". So I think property(3) is a non-starter.

Perhaps if the standard allows both your generator, and also my suspect generator above, and also allow properties to be visited in any order, we might only be able to guarantee the following:

(4) when we enter a particular iteration of the loop for(k in obj) it must be the case that at some moment since the loop started running there was some obj2 in the prototype chain of obj at that time (so obj2===obj or obj2===obj.proto and so on), such that obj2[k] was enumerable.

If that's the best we can do in the case when things are changing underfoot, we may want to explicitly state the properties we want when nothing has changed:

(5) when we enter a particular iteration of the loop for(k in obj): if for all obj2 in the prototype chain of obj (so obj2===obj or obj2===obj.proto and so on) it is the case that the prototype of obj2 has not been mutated since the loop began, and it is the case that obj2[k] has not been mutated since the loop began then it must be the case that obj[k] is enumerable.

If property(2) above is still too strong, then do you think properties (4) and (5) are ok? Or have I still not properly understood what you've told me?

I notice that even if property (2) is guaranteed by the ES5 spec, the introduction of Iterator objects in ES6 seem to make it less likely to be guaranteed there. Indeed, if I'm reading it correctly, the "informative algorithm" at the end of section 8.3.12 of the ES6 draft seems to violate property (2) above.

Sorry for the length of this email, and thanks for all your help so far!

Gareth.