Discuss
  • Proxy-induced impurity of internal methods

Proxy-induced impurity of internal methods

# Andreas Rossberg (14 years ago)

Proxies invalidate one fundamental assumption of the current ES spec, namely that (most) internal methods are effectively pure. That has a couple of consequences which the current proxy proposal and semantics seem to ignore, but which we need to address.

OBSERVABILITY & EFFICIENCY

In ES5, internal methods essentially are an implementation detail of the spec. AFAICS, there is no way their interaction is actually observable in user code. This gives JS implementations significant leeway in implementing objects (and they make use of it).

This changes drastically with proxies. In particular, since most internal methods may now invoke traps directly or indirectly, we can suddenly observe many internal steps of property lookup and similar operations through potential side effects of these traps. (Previously, only the invocation of getters or setters was observable).

Take the following simple example:

var desc = {configurable: true, get: function() {return 8}, set: function() {return true}} var handler = {getPropertyDescriptor: function() {seq += "G"; return desc}} var p = Proxy.create(handler) var o = Object.create(p)

var seq = "" o.x var seq1 = seq seq = "" o.x = 0 var seq2 = seq

According to the proxy spec, we should see seq1=="G" and seq2=="GG". In my local version of V8, I currently see seq1=="G" and seq2=="G". In Firefox 7, I see seq1=="GG" and seq2=="GG".

Obviously, both implementations are unfaithful to the spec, albeit in reverse ways. At least for V8, implementing the correct behaviour may require significant changes.

Also, I wonder whether the current semantics forcing seq2=="GG" really is what we want, given that it is unnecessarily inefficient (note that it also involves converting the property descriptor twice, which in turn can spawn numerous calls into user code). Optimizing this would require purity analysis on trap functions, which seems difficult in general.

HIDDEN ASSUMPTIONS

In a number of places, the ES5 spec makes hidden assumptions about the purity of internal method calls, and derives certain invariants from that, which break with proxies.

For example, in the spec of [[Put]] (8.12.5), step 5.a asserts that desc.[[Set]] cannot be undefined. That is true in ES5, but no longer with proxies. Unsurprisingly, both Firefox and V8 do funny things for the following example:

var handler = { getPropertyDescriptor: function() { Object.defineProperty(o, "x", {get: function() { return 5 }}) return {set: function() {}} } } var p = Proxy.create(handler) var o = Object.create(p) o.x = 4

Firefox 7: InternalError on line 1: too much recursion V8: TypeError: Trap #<error> of proxy handler #<Object> returned

non-configurable descriptor for property x

More generally, there is no guarantee anymore that the result of [[CanPut]] in step 1 of [[Put]] is in any way consistent with what we see in later steps. In this light (and due to the efficiency reasons I mentioned earlier), we might want to consider rethinking the CanPut/Put split.

This is just one case. There may be other problematic places in other operations. Most of them are probably more subtle, i.e. the spec still prescribes some behaviour, but that does not necessarily make any sense for certain cases (and would be hard to implement to the letter). We probably need to check the whole spec very carefully.

FIXING PROXIES

A particularly worrisome side effect is fixing a proxy. The proxy semantics contains a lot of places saying "If O is a trapping proxy, do steps I-J." However, there generally is no guarantee that O remains a trapping proxy through all of I-J!

Again, an example:

var handler = { get set() { Object.freeze(p); return undefined }, fix: function() { return {} } } var p = Proxy.create(handler) p.x

Firefox 7: TypeError on line 1: getPropertyDescriptor is not a function V8: TypeError: Object #<Object> has no method 'getPropertyDescriptor'

The current proxy semantics has an (informal) restriction forbidding reentrant fixing of the same object, but that is only a very special case of the broader problem. Firefox 7 rejects fixing a proxy while one of (most) its traps is executing (this seems to be a recent change, and the above case probably is an oversight). But it is not clear to me what the exact semantics is there, and whether it is enough as a restriction. V8 currently even crashes on a few contorted examples.

In summary, I'm slightly worried. The above all seems fixable, but is that all? Ideally, I'd like to see a more thorough analysis of how the addition of proxies affects properties of the language and its spec. But given the state of the ES spec, that is probably too much to wish for... :)

# Waldemar Horwat (14 years ago)

Please keep bringing these up; they're important.

This is something that we'll need to get nailed down for the spec. Yes, I'm worried too, as this problem is not well-understood. It has the feel of a research problem.

 Waldemar
# Andreas Rossberg (14 years ago)

On 5 October 2011 18:57, Andreas Rossberg <rossberg at google.com> wrote:

FIXING PROXIES

A particularly worrisome side effect is fixing a proxy. The proxy semantics contains a lot of places saying "If O is a trapping proxy, do steps I-J." However, there generally is no guarantee that O remains a trapping proxy through all of I-J!

Again, an example:

var handler = {  get set() { Object.freeze(p); return undefined },  fix: function() { return {} } } var p = Proxy.create(handler) p.x

Firefox 7: TypeError on line 1: getPropertyDescriptor is not a function V8: TypeError: Object #<Object> has no method 'getPropertyDescriptor'

Whoops, sorry, I just saw that I screwed up that example. That behaviour is perfectly fine, of course. Don't have my notes here, I'll deliver the proper example tomorrow.

# Allen Wirfs-Brock (14 years ago)

Good points that we will have to specify careful. Also one the reasons we do prototype implementations.

Such issues seems inherent in the adoption of an intercession API and semantics. Having to deal with such issues isn't really new. In ES5 we had to deal with this possibility WRT [[Get]] and [[Set]] triggering accessors it forced us in some cases to use [[DefineOwnProperty]] instead of [[Put]]. The impact on the internal methods were relatively small in ES5. It was larger for some of the syntactic production evaluation semantics. The biggest impact was on built-ins. I agree that Proxies will have even broader impact and I can even imagine that working through them might lead to some refactoring of the internal properties in order to reduce such effects. such refactorings might then force some changes in to the Proxy Traps.

more below

On Oct 5, 2011, at 9:57 AM, Andreas Rossberg wrote:

In summary, I'm slightly worried. The above all seems fixable, but is that all? Ideally, I'd like to see a more thorough analysis of how the addition of proxies affects properties of the language and its spec. But given the state of the ES spec, that is probably too much to wish for... :)

I'm not sure what you mean my the last sentence. I have not yet done any work to incorporate proxies into the ES6 draft. What we currently have is proposal that does not address this depth of specification. I can assure you that, we will when they start to move into the ES6 draft.

If you have specific issues like these a good way to capture them is to file bugs against the "proposals" component of the "harmony" products at bugs.ecmascript.org. Proposes resolutions would be good too. I definitely look at reported proposal bugs when I work on incorporating new features into the draft specification. On the other hand I don't guarantee that I will spot or remember all issues raised on this list. So file bugs.

# Andreas Rossberg (14 years ago)

On 6 October 2011 06:34, Allen Wirfs-Brock <allen at wirfs-brock.com> wrote:

On Oct 5, 2011, at 9:57 AM, Andreas Rossberg wrote:

In summary, I'm slightly worried. The above all seems fixable, but is that all? Ideally, I'd like to see a more thorough analysis of how the addition of proxies affects properties of the language and its spec. But given the state of the ES spec, that is probably too much to wish for... :)

I'm not sure what you mean my the last sentence. I have not yet done any work to incorporate proxies into the ES6 draft.

Oh, sorry, my remark was unintentionally ambiguous -- it wasn't directed at you. Just the generic rant that the whole ES spec is a horribly ad-hoc, utterly unanalysable beast using the state-of-the-art of language specification from 1960. :) Clearly nothing the editor could or should just fix at this point.

If you have specific issues like these a good way to capture them is to file bugs against  the "proposals" component  of the "harmony" products at bugs.ecmascript.org.  Proposes resolutions would be good too.  I definitely look at reported proposal bugs when I work on incorporating new features into the draft specification.  On the other hand I don't guarantee that I will spot or remember all issues raised on this list.  So file bugs.

Fair enough. This time, however, my comments were mainly meant for Tom & Mark, who are working on the proposal right now I think. I refrained from suggesting concrete fixes because they probably have a better idea what semantics they envision.

# Andreas Rossberg (14 years ago)

On 5 October 2011 21:00, Andreas Rossberg <rossberg at google.com> wrote:

On 5 October 2011 18:57, Andreas Rossberg <rossberg at google.com> wrote:

FIXING PROXIES

A particularly worrisome side effect is fixing a proxy. The proxy semantics contains a lot of places saying "If O is a trapping proxy, do steps I-J." However, there generally is no guarantee that O remains a trapping proxy through all of I-J!

Again, an example:

var handler = {  get set() { Object.freeze(p); return undefined },  fix: function() { return {} } } var p = Proxy.create(handler) p.x

Firefox 7: TypeError on line 1: getPropertyDescriptor is not a function V8: TypeError: Object #<Object> has no method 'getPropertyDescriptor'

Whoops, sorry, I just saw that I screwed up that example. That behaviour is perfectly fine, of course. Don't have my notes here, I'll deliver the proper example tomorrow.

Here we go (the last line should have been an assignment):

var handler = {  get set() { Object.freeze(p); return undefined },   fix: function() { return {} } } var p = Proxy.create(handler) p.x = 4

Firefox 7: TypeError on line 1: proxy was fixed while executing the handler V8: TypeError: Object #<Object> has no method 'getOwnPropertyDescriptor'

So Firefox rejects this (consistently with its treatment of other methods), while V8 tries to go on with the DefaultPut, using the traps from the handler that it still happens to have around. This is not quite what the rules of DefaultPut imply, but what the (inconsistent) note says.

A related nit: even for freeze and friends, the restriction on recursive fix is NOT enough as currently stated in the proxy semantics. Consider:

var handler = { get fix() { Object.seal(p); return {} } } var p = Proxy.create(handler) Object.freeze(p)

Strictly speaking, there actually is no recursive execution of fix() -- the recursion occurs a few steps earlies, when we try to get the fix function. Firefox rejects this nevertheless:

TypeError on line 2: proxy was fixed while executing the handler

V8 bails out with a stack overflow:

RangeError: Maximum call stack size exceeded

While this might merely be a nit, it shows that it is not generally enough to only prevent fixing while executing traps. To be conservative, it seems like we perhaps have to disallow any reentrant use of freeze/seal/prevenExt at any point in any internal method of the same object. But how spec that?

# Tom Van Cutsem (14 years ago)

These are all good points. Some comments below:

2011/10/5 Andreas Rossberg <rossberg at google.com>

Proxies invalidate one fundamental assumption of the current ES spec, namely that (most) internal methods are effectively pure. That has a couple of consequences which the current proxy proposal and semantics seem to ignore, but which we need to address.

OBSERVABILITY & EFFICIENCY

In ES5, internal methods essentially are an implementation detail of the spec. AFAICS, there is no way their interaction is actually observable in user code. This gives JS implementations significant leeway in implementing objects (and they make use of it).

This changes drastically with proxies. In particular, since most internal methods may now invoke traps directly or indirectly, we can suddenly observe many internal steps of property lookup and similar operations through potential side effects of these traps. (Previously, only the invocation of getters or setters was observable).

Take the following simple example:

var desc = {configurable: true, get: function() {return 8}, set: function() {return true}} var handler = {getPropertyDescriptor: function() {seq += "G"; return desc}} var p = Proxy.create(handler) var o = Object.create(p)

var seq = "" o.x var seq1 = seq seq = "" o.x = 0 var seq2 = seq

According to the proxy spec, we should see seq1=="G" and seq2=="GG". In my local version of V8, I currently see seq1=="G" and seq2=="G". In Firefox 7, I see seq1=="GG" and seq2=="GG".

This point was previously noted, see: < strawman:proxy_set_trap>

It was brought up at the March 2011 meeting, and IIRC we were in agreement that the spec. should be adapted to remove the redundant getPropertyDescriptor call.

Obviously, both implementations are unfaithful to the spec, albeit in reverse ways. At least for V8, implementing the correct behaviour may require significant changes.

Also, I wonder whether the current semantics forcing seq2=="GG" really is what we want, given that it is unnecessarily inefficient (note that it also involves converting the property descriptor twice, which in turn can spawn numerous calls into user code). Optimizing this would require purity analysis on trap functions, which seems difficult in general.

I agree. This is clearly a case where the ES5 spec was written assuming this was all unobservable.

HIDDEN ASSUMPTIONS

In a number of places, the ES5 spec makes hidden assumptions about the purity of internal method calls, and derives certain invariants from that, which break with proxies.

For example, in the spec of [[Put]] (8.12.5), step 5.a asserts that desc.[[Set]] cannot be undefined. That is true in ES5, but no longer with proxies. Unsurprisingly, both Firefox and V8 do funny things for the following example:

var handler = { getPropertyDescriptor: function() { Object.defineProperty(o, "x", {get: function() { return 5 }}) return {set: function() {}} } } var p = Proxy.create(handler) var o = Object.create(p) o.x = 4

Firefox 7: InternalError on line 1: too much recursion V8: TypeError: Trap #<error> of proxy handler #<Object> returned non-configurable descriptor for property x

(are you sure this tests the right behavior? It seems the V8 TypeError is simply due to the fact that the descriptor returned from getPropertyDescriptor is configurable.)

More generally, there is no guarantee anymore that the result of

[[CanPut]] in step 1 of [[Put]] is in any way consistent with what we see in later steps. In this light (and due to the efficiency reasons I mentioned earlier), we might want to consider rethinking the CanPut/Put split.

I agree. In fact, proxies already abandon the CanPut/Put split: they implement CanPut simply by always returning true, and perform all of their assignment logic in [[Put]]. Related to this refactoring: Mark has previously proposed introducing a [[Set]] trap that simply returns a boolean, indicating whether or not the assignment succeeded. The [[Put]] trap would simply call [[Set]], converting a false result into a TypeError when appropriate (cf. < harmony:proxy_defaulthandler#alternative_implementation_for_default_set_trap>).

We don't have consensus on this yet. I would propose to discuss the CanPut/Put refactoring and the [[Set]] alternative together during the Nov. meeting.

This is just one case. There may be other problematic places in other operations. Most of them are probably more subtle, i.e. the spec still prescribes some behaviour, but that does not necessarily make any sense for certain cases (and would be hard to implement to the letter). We probably need to check the whole spec very carefully.

FIXING PROXIES

A particularly worrisome side effect is fixing a proxy. The proxy semantics contains a lot of places saying "If O is a trapping proxy, do steps I-J." However, there generally is no guarantee that O remains a trapping proxy through all of I-J!

Again, an example:

var handler = { get set() { Object.freeze(p); return undefined }, fix: function() { return {} } } var p = Proxy.create(handler) p.x

Firefox 7: TypeError on line 1: getPropertyDescriptor is not a function V8: TypeError: Object #<Object> has no method 'getPropertyDescriptor'

The current proxy semantics has an (informal) restriction forbidding reentrant fixing of the same object, but that is only a very special case of the broader problem. Firefox 7 rejects fixing a proxy while one of (most) its traps is executing (this seems to be a recent change, and the above case probably is an oversight). But it is not clear to me what the exact semantics is there, and whether it is enough as a restriction. V8 currently even crashes on a few contorted examples.

You are right, the restriction on recursive fixing should be generalized such that fixing is disallowed as long as there are any remaining active trap invocations, unless we can make sure that fixing in the middle of any step leads to a sane outcome.

On the upside: Mark and I have been making good progress on an alternative proxy proposal that completely decouples Object.{freeze,seal,preventExtensions} from fixing a proxy. That would mean that Object.{freeze,seal,preventExtensions} would no longer suffer from the above restriction - they don't implicitly fix the proxy anymore. There would be a distinct Object.stopTrapping operation that cannot be performed while the proxy is actively trapping. That would confine the problem to this (much more specific) operation.

# Tom Van Cutsem (14 years ago)

2011/10/6 Andreas Rossberg <rossberg at google.com>

A related nit: even for freeze and friends, the restriction on recursive fix is NOT enough as currently stated in the proxy semantics. Consider:

var handler = { get fix() { Object.seal(p); return {} } } var p = Proxy.create(handler) Object.freeze(p)

Strictly speaking, there actually is no recursive execution of fix() -- the recursion occurs a few steps earlies, when we try to get the fix function. Firefox rejects this nevertheless:

TypeError on line 2: proxy was fixed while executing the handler

V8 bails out with a stack overflow:

RangeError: Maximum call stack size exceeded

While this might merely be a nit, it shows that it is not generally enough to only prevent fixing while executing traps. To be conservative, it seems like we perhaps have to disallow any reentrant use of freeze/seal/prevenExt at any point in any internal method of the same object. But how spec that?

In my FixedHandler code, I used the following trick to disallow recursive fixing:

// this fix function is a wrapper around the 'real' fix function: fix: function(operation) { if (this.fixing) { throw new TypeError("cannot recursively call the fix() trap while fixing a proxy"); } var props = null; try { this.fixing = true; props = this.targetHandler.fix(operation); } finally { delete this.fixing; }

// process props ...

},

I guess Firefox does something similar in spirit. Since the flag is set before the lookup of this.targetHandler.fix, that explains why even during the lookup of "fix", the proxy can't fix recursively. Generalizing the flag to account for all traps is going to be tricky and expensive, though, which suggests that this may not be the right behavior.

I will go over the proposed proxies spec to check whether there is actually any harm in allowing a proxy to become non-trapping during an active trap. If the proxy describes a coherent object before and after the state change, there is no reason to disallow this. The new proposal Mark and I have been working on may help here, since it enforces more invariants on proxies.

# Andreas Rossberg (14 years ago)

On 10 October 2011 15:38, Tom Van Cutsem <tomvc.be at gmail.com> wrote:

This point was previously noted, see: strawman:proxy_set_trap It was brought up at the March 2011 meeting, and IIRC we were in agreement that the spec. should be adapted to remove the redundant getPropertyDescriptor call.

Ah, thanks. I wasn't aware of that. Good to hear.

In a number of places, the ES5 spec makes hidden assumptions about the purity of internal method calls, and derives certain invariants from that, which break with proxies.

For example, in the spec of [[Put]] (8.12.5), step 5.a asserts that desc.[[Set]] cannot be undefined. That is true in ES5, but no longer with proxies. Unsurprisingly, both Firefox and V8 do funny things for the following example:

var handler = {  getPropertyDescriptor: function() {    Object.defineProperty(o, "x", {get: function() { return 5 }})    return {set: function() {}}  } } var p = Proxy.create(handler) var o = Object.create(p) o.x = 4

Firefox 7: InternalError on line 1: too much recursion V8: TypeError: Trap #<error> of proxy handler #<Object> returned non-configurable descriptor for property x

(are you sure this tests the right behavior? It seems the V8 TypeError is simply due to the fact that the descriptor returned from getPropertyDescriptor is configurable.)

You are right, of course. If I make it configurable, V8 returns without error. However, by modifying the example somewhat you can see that it executes the setter from the descriptor then. That is not quite right either (Though neither wrong, I suppose :) ).

I agree. In fact, proxies already abandon the CanPut/Put split: they implement CanPut simply by always returning true, and perform all of their assignment logic in [[Put]]. Related to this refactoring: Mark has previously proposed introducing a [[Set]] trap that simply returns a boolean, indicating whether or not the assignment succeeded. The [[Put]] trap would simply call [[Set]], converting a false result into a TypeError when appropriate (cf. harmony:proxy_defaulthandler#alternative_implementation_for_default_set_trap). We don't have consensus on this yet. I would propose to discuss the CanPut/Put refactoring and the [[Set]] alternative together during the Nov. meeting.

Yes, that makes sense.

On 10 October 2011 16:01, Tom Van Cutsem <tomvc.be at gmail.com> wrote:

I will go over the proposed proxies spec to check whether there is actually any harm in allowing a proxy to become non-trapping during an active trap. If the proxy describes a coherent object before and after the state change, there is no reason to disallow this. The new proposal Mark and I have been working on may help here, since it enforces more invariants on proxies.

I'm not sure I understand what you mean by "becoming non-trapping", can you elaborate? What would it do instead?

# Tom Van Cutsem (14 years ago)

2011/10/10 Andreas Rossberg <rossberg at google.com>

On 10 October 2011 16:01, Tom Van Cutsem <tomvc.be at gmail.com> wrote:

I will go over the proposed proxies spec to check whether there is actually any harm in allowing a proxy to become non-trapping during an active trap. If the proxy describes a coherent object before and after the state change, there is no reason to disallow this. The new proposal Mark and I have been working on may help here, since it enforces more invariants on proxies.

I'm not sure I understand what you mean by "becoming non-trapping", can you elaborate? What would it do instead?

By becoming non-trapping, I simply meant "fixing the proxy" (i.e. the state where the proxy no longer traps the handler, but instead "becomes" a normal Object).

In the proposal we are writing up, we would split up the old fix trap in two different traps:

  • Object.{freeze,seal,preventExtensions}(p) would trigger p's "protect" trap. That trap would effectively make the proxy frozen/sealed/non-extensible. A frozen/sealed/non-extensible proxy can still be trapping.
  • A separate Proxy.stopTrapping(p) function would trigger p's "stopTrapping" trap. If that trap does not reject the request, the proxy handler would be "switched off", effectively turning the proxy into a regular object. This corresponds to the "become" behavior of the current fix() trap.
# Tom Van Cutsem (14 years ago)

2011/10/6 Andreas Rossberg <rossberg at google.com>

A related nit: even for freeze and friends, the restriction on recursive fix is NOT enough as currently stated in the proxy semantics. Consider:

var handler = { get fix() { Object.seal(p); return {} } } var p = Proxy.create(handler) Object.freeze(p)

Strictly speaking, there actually is no recursive execution of fix() -- the recursion occurs a few steps earlies, when we try to get the fix function. Firefox rejects this nevertheless:

TypeError on line 2: proxy was fixed while executing the handler

V8 bails out with a stack overflow:

RangeError: Maximum call stack size exceeded

While this might merely be a nit, it shows that it is not generally enough to only prevent fixing while executing traps. To be conservative, it seems like we perhaps have to disallow any reentrant use of freeze/seal/prevenExt at any point in any internal method of the same object. But how spec that?

I've been re-examining this issue for direct proxies. My conclusion is that for direct proxies, in principle, stopTrapping can safely be invoked while executing any trap. The reason why this works is primarily because direct proxies, as currently proposed, no longer need default implementations for missing derived traps.

The first thing an intercepted operation does is to [[Get]] the trap from the handler, and to [[Call]] it if it exists. Once the trap is [[Call]]-ed, the handler is never referenced anymore. So, if a proxy becomes non-trapping with active trap invocations still on the stack, those trap invocations, when they resume, will never reference the handler anymore, so are not critically affected by the fact that the proxy is now "fixed" and the handler is nulled out.

The above example (trying to fix a proxy while fixing it) would then go into an infinite loop, which is perfectly reasonable from a semantic point of view. It's no different from a handler trying to implement getOwnPropertyDescriptor by calling getOwnPropertyDescriptor on the proxy itself.

Whether or not we should re-check the state of a proxy after [[Get]]-ting a trap implemented as an accessor remains unclear. For direct proxies, either option is safe, but we have to make a choice since the difference is observable, cf. < code.google.com/p/es-lab/source/browse/trunk/src/proxies/DirectProxies.js#363