Discuss
  • [Harmony Proxies] LazyReadCopy experiment and invariant checking for [[Extensible]]=false

[Harmony Proxies] LazyReadCopy experiment and invariant checking for [[Extensible]]=false

# David Bruant (14 years ago)

Recently, I've been thinking about the structured clone algorithm used in postMessage The browser has to do a copy of an object and pass it to the message receiving context. In some cases, if the object has a lot of properties, it could be costly to do this copy. So I thought "hey, what about wrapping it with a proxy and just hand the proxy?". For the rest of this use case, I will consider myself in the role of the browser (implementing postMessage and tryiing to do so in JavaScript) and I may use the "become" primitive if needed since I'm in "privileged code".

Basically, the proxy has to forward "reads" to the target object, keep the "writes" internally and make sure future "reads" are consistent if some "writes" occured. You can see my implementation of "LazyReadCopy" at: DavidBruant/HarmonyProxyLab/tree/master/LazyReadCopy (I discuss limitations below)

So now we can send an object in the message Receiving Context (RC). This object is a copy of the one in the message Sending Context (SC). The RC can manipulate the object and this will have no noticeable effect in the SC. However, the opposite is not true. If a property is added in the SC, the RC can read this new property. So, for my implementation to work, I would need 2 lazy read copies. One for the SC, one for the RC. The one in the SC can replace the object thanks to the "become" privileged primitive.

So far, so good, we have an object, two lazy read copies of it (one in each context), basic expectations are respected.

Now, consider the following snippet:

var o = {}; Object.preventExtension(o); Object.isExtensible(o); // false receivingContext.postMessage(o); // The browser replaces o with a lazy read copy of o. That's now a proxy. Object.isExtensible(o); // true, because it's a proxy :-s

Of course, that's my fault, I shouldn't play too much with the "become" primitive. However I feel that I should be allowed to create non-extensible proxies to cover this use case.

And well, you probably see my proposal coming: what about doing invariant-enforcing with [[Extensible]]=false instead of the fix+become? There would only be a need for a preventExtension fundamental trap (freeze and seal would be derived or even not trap at all, calling the preventExtension trap anyway)

I haven't taken the time to try to implement it, but starting from Tom's FixedHandler implementation [1], it looks like it would be just adding a this.extensible boolean (and the set of properties can be retrieved from this.fixedProps). One-property traps would need a lookup to a property set (hopefully, JS engines have efficient implementations for that for the "in" operator) to avoid lies on whether the property is part of the object already or not. The rest of the one-property traps invariants are already taken care of by FixedHandler. Several-property traps (get{Own}PropertyNames, enumerate, keys) could call the trap, throw away the result and return the set of properties. The inconvenient of this would be that the user-returned order wouldn't be respected. The costly alternative is to check that the returned array isn't adding properties that shouldn't be there.

Another idea is to provide the choice between the invariant-checking flavor and the "turn me into a normal object" flavor with the return value of the fix/preventExtension trap.

Any thoughts?

Limitations of my implementation regarding the structured clone algorithm.

  • Non-enumerable and inherited properties are not filtered out
  • Doing a lazy copy read is not performed recursively
  • The object in the receiving context should only have data property, so if there are accessors, their getters should be called once at copy time and the result stored in the copy. (this last point cannot be implemented efficiently in JavaScript, I think)

David

[1] code.google.com/p/es-lab/source/browse/trunk/src/proxies/FixedHandler.js

# Tom Van Cutsem (14 years ago)

2011/7/13 David Bruant <david.bruant at labri.fr>

Hi,

Recently, I've been thinking about the structured clone algorithm used in postMessage The browser has to do a copy of an object and pass it to the message receiving context. In some cases, if the object has a lot of properties, it could be costly to do this copy. So I thought "hey, what about wrapping it with a proxy and just hand the proxy?".

This is a good use-case, and definitely one that Mark and I would like to explore with Proxies, as indicated by the "Eventual reference proxy" example on the harmony:proxies wiki page. Also, it's not just that copying objects is sometimes too costly, sometimes it's just not what the programmer wants. If the passed object is mutable, you don't want to create copies that can then diverge when e.g. distributed to separate web workers. Also, if the object represents some local resource that is unavailable to the receiver endpoint, you want to hand it a reference to the service, not a copy of some data.

For the rest of this use case, I will consider myself in the role of the browser (implementing postMessage and tryiing to do so in JavaScript) and I may use the "become" primitive if needed since I'm in "privileged code".

Basically, the proxy has to forward "reads" to the target object, keep the "writes" internally and make sure future "reads" are consistent if some "writes" occured. You can see my implementation of "LazyReadCopy" at: DavidBruant/HarmonyProxyLab/tree/master/LazyReadCopy (I discuss limitations below)

I'm having trouble relating the "LazyReadCopy" proxy to your above use case. If I were to create a proxy that represents a reference to an object in another frame/web worker, I wouldn't cache writes in the proxy. I'm not saying LazyReadCopy never makes sense, it just seems to make less sense in the context you describe.

So now we can send an object in the message Receiving Context (RC). This object is a copy of the one in the message Sending Context (SC). The RC can manipulate the object and this will have no noticeable effect in the SC. However, the opposite is not true. If a property is added in the SC, the RC can read this new property. So, for my implementation to work, I would need 2 lazy read copies. One for the SC, one for the RC. The one in the SC can replace the object thanks to the "become" privileged primitive.

Why the need for "become"? Why does the SC need to refer to a proxy, rather than to the real target object directly?

So far, so good, we have an object, two lazy read copies of it (one in each context), basic expectations are respected.

Now, consider the following snippet:

var o = {}; Object.preventExtension(o); Object.isExtensible(o); // false receivingContext.postMessage(o); // The browser replaces o with a lazy read copy of o. That's now a proxy. Object.isExtensible(o); // true, because it's a proxy :-s

Of course, that's my fault, I shouldn't play too much with the "become" primitive. However I feel that I should be allowed to create non-extensible proxies to cover this use case.

Again, why the need for "become" here? We're in the sending context, which already has direct access to "o". No need to wrap it here.

And well, you probably see my proposal coming: what about doing invariant-enforcing with [[Extensible]]=false instead of the fix+become? There would only be a need for a preventExtension fundamental trap (freeze and seal would be derived or even not trap at all, calling the preventExtension trap anyway)

I haven't taken the time to try to implement it, but starting from Tom's FixedHandler implementation [1], it looks like it would be just adding a this.extensible boolean (and the set of properties can be retrieved from this.fixedProps).

I'll have to think about it. It may be that with the FixedHandler infrastructure in place, supporting [[Extensible]]:true proxies falls out naturally. But it might as well impose a burden on every single handler to explicitly deal with the extensible vs non-extensible state, which would be a high price to pay.

We'll still need a fix() trap so that the FixedHandler can ask the real handler for all of its configurable properties as well, although those could also be derived by calling getOwnPropertyNames() + getOwnPropertyDescriptor().

One-property traps would need a lookup to a property set (hopefully, JS engines have efficient implementations for that for the "in" operator) to avoid lies on whether the property is part of the object already or not. The rest of the one-property traps invariants are already taken care of by FixedHandler. Several-property traps (get{Own}PropertyNames, enumerate, keys) could call the trap, throw away the result and return the set of properties. The inconvenient of this would be that the user-returned order wouldn't be respected. The costly alternative is to check that the returned array isn't adding properties that shouldn't be there.

If the number of properties doesn't match up, that would be a quick test to detect when the trap lies. Unfortunately that only tests for the supposedly uncommon case.

Another idea is to provide the choice between the invariant-checking

flavor and the "turn me into a normal object" flavor with the return value of the fix/preventExtension trap.

I would prefer one mechanism over two. If the FixedHandler can be extended with reasonable checks to uphold all invariants related to Object.{preventExtensions|seal|freeze}, I don't see a need for the fix+become mechanism anymore.

# David Bruant (14 years ago)

Le 14/07/2011 16:23, Tom Van Cutsem a écrit :

Hi David,

2011/7/13 David Bruant <david.bruant at labri.fr <mailto:david.bruant at labri.fr>>

Hi,

Recently, I've been thinking about the structured clone algorithm used
in postMessage
The browser has to do a copy of an object and pass it to the message
receiving context. In some cases, if the object has a lot of
properties,
it could be costly to do this copy. So I thought "hey, what about
wrapping it with a proxy and just hand the proxy?".

This is a good use-case, and definitely one that Mark and I would like to explore with Proxies, as indicated by the "Eventual reference proxy" example on the harmony:proxies wiki page. Also, it's not just that copying objects is sometimes too costly, sometimes it's just not what the programmer wants. If the passed object is mutable, you don't want to create copies that can then diverge when e.g. distributed to separate web workers.

You need to create copies, don't you? Otherwise I think you introduce non-determinism since web workers can have parallel execution. That's the point of having a structured clone algorithm I thought. Having mutable shared data across different contexts (potentially running in parallel) is prevented, isn't it?

Also, if the object represents some local resource that is unavailable to the receiver endpoint, you want to hand it a reference to the service, not a copy of some data.

For the rest of this use case, I will consider myself in the role
of the
browser (implementing postMessage and tryiing to do so in JavaScript)
and I may use the "become" primitive if needed since I'm in
"privileged
code".

Basically, the proxy has to forward "reads" to the target object, keep
the "writes" internally and make sure future "reads" are consistent if
some "writes" occured.
You can see my implementation of "LazyReadCopy" at:
https://github.com/DavidBruant/HarmonyProxyLab/tree/master/LazyReadCopy
(I discuss limitations below)

I'm having trouble relating the "LazyReadCopy" proxy to your above use case. If I were to create a proxy that represents a reference to an object in another frame/web worker, I wouldn't cache writes in the proxy. I'm not saying LazyReadCopy never makes sense, it just seems to make less sense in the context you describe.

My intention is to give to the two contexts the impression they're manipulating two different objects (because in postMessage, the message in the receiving and sending contexts are two different objects)

So now we can send an object in the message Receiving Context
(RC). This
object is a copy of the one in the message Sending Context (SC).
The RC
can manipulate the object and this will have no noticeable effect
in the SC.
However, the opposite is not true. If a property is added in the
SC, the
RC can read this new property. So, for my implementation to work, I
would need 2 lazy read copies. One for the SC, one for the RC. The one
in the SC can replace the object thanks to the "become" privileged
primitive.

Why the need for "become"? Why does the SC need to refer to a proxy, rather than to the real target object directly?

Consider:

// In sending context var o = {}; receivingContext.postMessage(o); o.a = 1; // after having sent the message

at no time the RC should be able to observe an 'a' property. The message was an empty object, so that's what the RC should observe (at least in the postMessage use case). Yet, I have no reason to prevent the SC to change o after a postMessage, so in order to keep the reads consistent and still allow writes, I can turn my object into a LazyReadCopy object and do the become.

So far, so good, we have an object, two lazy read copies of it (one in
each context), basic expectations are respected.

Now, consider the following snippet:
-----
var o = {};
Object.preventExtension(o);
Object.isExtensible(o); // false
receivingContext.postMessage(o);
// The browser replaces o with a lazy read copy of o. That's now a
proxy.
Object.isExtensible(o); // true, because it's a proxy :-s
-----
Of course, that's my fault, I shouldn't play too much with the
"become"
primitive.
However I feel that I should be allowed to create non-extensible
proxies
to cover this use case.

Again, why the need for "become" here? We're in the sending context, which already has direct access to "o". No need to wrap it here.

And well, you probably see my proposal coming: what about doing
invariant-enforcing with [[Extensible]]=false instead of the
fix+become?
There would only be a need for a preventExtension fundamental trap
(freeze and seal would be derived or even not trap at all, calling the
preventExtension trap anyway)

I haven't taken the time to try to implement it, but starting from
Tom's
FixedHandler implementation [1], it looks like it would be just
adding a
this.extensible boolean (and the set of properties can be
retrieved from
this.fixedProps).

I'll have to think about it. It may be that with the FixedHandler infrastructure in place, supporting [[Extensible]]:true proxies falls out naturally. But it might as well impose a burden on every single handler to explicitly deal with the extensible vs non-extensible state, which would be a high price to pay.

We'll still need a fix() trap so that the FixedHandler can ask the real handler for all of its configurable properties as well, although those could also be derived by calling getOwnPropertyNames() + getOwnPropertyDescriptor().

One-property traps would need a lookup to a property set
(hopefully, JS
engines have efficient implementations for that for the "in" operator)
to avoid lies on whether the property is part of the object already or
not. The rest of the one-property traps invariants are already taken
care of by FixedHandler.
Several-property traps (get{Own}PropertyNames, enumerate, keys) could
call the trap, throw away the result and return the set of properties.
The inconvenient of this would be that the user-returned order
wouldn't
be respected. The costly alternative is to check that the returned
array
isn't adding properties that shouldn't be there.

If the number of properties doesn't match up, that would be a quick test to detect when the trap lies. Unfortunately that only tests for the supposedly uncommon case.

Another idea is to provide the choice between the invariant-checking
flavor and the "turn me into a normal object" flavor with the return
value of the fix/preventExtension trap.

I would prefer one mechanism over two. If the FixedHandler can be extended with reasonable checks to uphold all invariants related to Object.{preventExtensions|seal|freeze}, I don't see a need for the fix+become mechanism anymore.

Good to know. I'm working on an implementation. It will be independent from the FixedHandler one. It should be possible to combine both easily though. I'll give a follow-up here when I'm done.

# David Bruant (14 years ago)
  • Mark to discuss ES5.1 invariants.

I'm working on an implementation. It will be independent from the FixedHandler one. It should be possible to combine both easily though.

I'll give a follow-up here when I'm done.

So, here it is: DavidBruant/HarmonyProxyLab/tree/master/NonExtensibleProxies I have just commited but not tested. The reason is that I wondered if all this effort was worth. I've re-read the invariants in ES5 (last one on extensible not allowed to go from false to true omitted):

  • If the value of the host object’s [[Extensible]] internal property is has been observed by ECMAScript code to be false, then if a call to [[GetOwnProperty]] describes a property as non-existent all subsequent calls must also describe that property as non-existent. => To implement this, the handler just has to keep a record of "already

observed as non-existent properties" and make sure [[GetOwnProperty]] respects the invariant by a lookup in this record.

  • The [[DefineOwnProperty]] internal method of a host object must not permit the addition of a new property to a host object if the [[Extensible]] internal property of that host object has been observed by ECMAScript code to be false. => This one is a bit more tricky. "must not permit the addition of a new

property" seems to assume that an object has a set of properties. But that's not really true for proxies. The engine has no way to say whether a property is defined in a proxy or not. Consequently, it cannot tell if [[DefineOwnProperty]] is called on a "new" or "old" property. This could be compensated by explicitely returning a list of property names in the fix/preventExtension trap.

Also, none of the 3 invariants regarding [[Extensible]]=false are about property names enumeration. Is there just no invariant to hold on enumeration? If there is none, well, no need to do O(n)-ish enforcements!

I'll update and test my code based on conclusions of this thread.

# Tom Van Cutsem (14 years ago)

2011/7/14 David Bruant <david.bruant at labri.fr>

Le 14/07/2011 16:23, Tom Van Cutsem a écrit :

Hi David,

2011/7/13 David Bruant <david.bruant at labri.fr>

Hi,

Recently, I've been thinking about the structured clone algorithm used in postMessage The browser has to do a copy of an object and pass it to the message receiving context. In some cases, if the object has a lot of properties, it could be costly to do this copy. So I thought "hey, what about wrapping it with a proxy and just hand the proxy?".

This is a good use-case, and definitely one that Mark and I would like to explore with Proxies, as indicated by the "Eventual reference proxy" example on the harmony:proxies wiki page. Also, it's not just that copying objects is sometimes too costly, sometimes it's just not what the programmer wants. If the passed object is mutable, you don't want to create copies that can then diverge when e.g. distributed to separate web workers.

You need to create copies, don't you? Otherwise I think you introduce non-determinism since web workers can have parallel execution. That's the point of having a structured clone algorithm I thought. Having mutable shared data across different contexts (potentially running in parallel) is prevented, isn't it?

Well, in the particular case of the eventual reference abstraction, any operations with side-effects that the RC performs on the original object are queued as separate events in the SC's event queue. Only the SC has direct (synchronous) access to the object. The RC doesn't have a local copy of the object, just a unique reference that it can use to send messages (in this particular case, asynchronous messages), to the target object. At no point can the RC gain synchronous access to the target. That avoids the shared-state concurrency quagmire.

Also, if the object represents some local resource that is unavailable to

the receiver endpoint, you want to hand it a reference to the service, not a copy of some data.

For the rest of this use case, I will consider myself in the role of the browser (implementing postMessage and tryiing to do so in JavaScript) and I may use the "become" primitive if needed since I'm in "privileged code".

Basically, the proxy has to forward "reads" to the target object, keep the "writes" internally and make sure future "reads" are consistent if some "writes" occured. You can see my implementation of "LazyReadCopy" at: DavidBruant/HarmonyProxyLab/tree/master/LazyReadCopy (I discuss limitations below)

I'm having trouble relating the "LazyReadCopy" proxy to your above use case. If I were to create a proxy that represents a reference to an object in another frame/web worker, I wouldn't cache writes in the proxy. I'm not saying LazyReadCopy never makes sense, it just seems to make less sense in the context you describe.

My intention is to give to the two contexts the impression they're manipulating two different objects (because in postMessage, the message in the receiving and sending contexts are two different objects)

So now we can send an object in the message Receiving Context (RC). This object is a copy of the one in the message Sending Context (SC). The RC can manipulate the object and this will have no noticeable effect in the SC. However, the opposite is not true. If a property is added in the SC, the RC can read this new property. So, for my implementation to work, I would need 2 lazy read copies. One for the SC, one for the RC. The one in the SC can replace the object thanks to the "become" privileged primitive.

Why the need for "become"? Why does the SC need to refer to a proxy, rather than to the real target object directly?

Consider:

// In sending context var o = {}; receivingContext.postMessage(o); o.a = 1; // after having sent the message

at no time the RC should be able to observe an 'a' property. The message was an empty object, so that's what the RC should observe (at least in the postMessage use case). Yet, I have no reason to prevent the SC to change o after a postMessage, so in order to keep the reads consistent and still allow writes, I can turn my object into a LazyReadCopy object and do the become.

Ok, thanks for clarifying. I now understand your case better. When I think about passing "o" to RC, if "o" is mutable, what I really want is for RC to receive a remote reference to "o", not a copy of "o". But let's not digress too far into the details of parameter-passing semantics.

# Tom Van Cutsem (14 years ago)

2011/7/14 David Bruant <david.bruant at labri.fr>

  • Mark to discuss ES5.1 invariants.

I'm working on an implementation. It will be independent from the FixedHandler one. It should be possible to combine both easily though.

I'll give a follow-up here when I'm done. So, here it is:

DavidBruant/HarmonyProxyLab/tree/master/NonExtensibleProxies I have just commited but not tested. The reason is that I wondered if all this effort was worth. I've re-read the invariants in ES5 (last one on extensible not allowed to go from false to true omitted):

  • If the value of the host object’s [[Extensible]] internal property is has been observed by ECMAScript code to be false, then if a call to [[GetOwnProperty]] describes a property as non-existent all subsequent calls must also describe that property as non-existent. => To implement this, the handler just has to keep a record of "already observed as non-existent properties" and make sure [[GetOwnProperty]] respects the invariant by a lookup in this record.

Which is bad since the set of "already observed as non-existent properties" is a potentially unbounded set determined by clients. We wouldn't want a proxy to have to remember all property names for which it ever replied "undefined".

  • The [[DefineOwnProperty]] internal method of a host object must not permit the addition of a new property to a host object if the [[Extensible]] internal property of that host object has been observed by ECMAScript code to be false. => This one is a bit more tricky. "must not permit the addition of a new property" seems to assume that an object has a set of properties. But that's not really true for proxies. The engine has no way to say whether a property is defined in a proxy or not. Consequently, it cannot tell if [[DefineOwnProperty]] is called on a "new" or "old" property. This could be compensated by explicitely returning a list of property names in the fix/preventExtension trap.

Indeed. I think it's still necessary that upon fixing a proxy, the proxy asks the handler for all of its own properties (either via the return value of fix() or by calling getOwnPropertyNames + getOwnPropertyDescriptor). That gives it a fixed set of properties, making it at least easy to check the previous invariant without requiring unbounded storage (which also seems to be the way you have implemented it in your prototype, I noticed).

Not sure how to deal with inherited properties, though. A fixed proxy may still have a non-fixed [[Prototype]], so its [[GetProperty]] and [[Get]] traps may still report new properties. OTOH, if the proxy's entire [[Prototype]] chain is also fixed, the constraints become more tight. A compromise could be that a non-extensible proxy can no longer virtualize inherited properties, only own properties.

# David Bruant (14 years ago)

Le 15/07/2011 11:44, Tom Van Cutsem a écrit :

2011/7/14 David Bruant <david.bruant at labri.fr <mailto:david.bruant at labri.fr>>

+ Mark to discuss ES5.1 invariants.

> I'm working on an implementation. It will be independent from the
> FixedHandler one. It should be possible to combine both easily
though.
>
> I'll give a follow-up here when I'm done.
So, here it is:
https://github.com/DavidBruant/HarmonyProxyLab/tree/master/NonExtensibleProxies
I have just commited but not tested. The reason is that I wondered if
all this effort was worth.
I've re-read the invariants in ES5 (last one on extensible not allowed
to go from false to true omitted):
* If the value of the host object’s [[Extensible]] internal
property is
has been observed by ECMAScript code to be false, then if a call to
[[GetOwnProperty]] describes a property as non-existent all subsequent
calls must also describe that property as non-existent.
=> To implement this, the handler just has to keep a record of
"already
observed as non-existent properties" and make sure [[GetOwnProperty]]
respects the invariant by a lookup in this record.

Which is bad since the set of "already observed as non-existent properties" is a potentially unbounded set determined by clients. We wouldn't want a proxy to have to remember all property names for which it ever replied "undefined".

True, I didn't think of that.

* The [[DefineOwnProperty]] internal method of a host object must not
permit the addition of a new property to a host object if the
[[Extensible]] internal property of that host object has been observed
by ECMAScript code to be false.
=> This one is a bit more tricky. "must not permit the addition of
a new
property" seems to assume that an object has a set of properties. But
that's not really true for proxies. The engine has no way to say
whether
a property is defined in a proxy or not. Consequently, it cannot
tell if
[[DefineOwnProperty]] is called on a "new" or "old" property.
This could be compensated by explicitely returning a list of property
names in the fix/preventExtension trap.

Indeed. I think it's still necessary that upon fixing a proxy, the proxy asks the handler for all of its own properties (either via the return value of fix() or by calling getOwnPropertyNames + getOwnPropertyDescriptor).

The return value of fix() sounds closer to the previous design. Moreover, there is no need to get the property descriptor. Non-extensible objects invariants are just about the properties regardless of their property descriptor. Also, in order to keep respecting non-configurable properties invariants, it should be said that non-configurable property names are appended to the return value of the fix trap.

That gives it a fixed set of properties, making it at least easy to check the previous invariant without requiring unbounded storage (which also seems to be the way you have implemented it in your prototype, I noticed).

Not sure how to deal with inherited properties, though.

Does it need to be addressed? Are there invariants requiring to enforce something on how to handle inheritance?

A fixed proxy may still have a non-fixed [[Prototype]], so its [[GetProperty]] and [[Get]] traps may still report new properties. OTOH, if the proxy's entire [[Prototype]] chain is also fixed, the constraints become more tight. A compromise could be that a non-extensible proxy can no longer virtualize inherited properties, only own properties.

Something like the inheritance-safe proxy idea [1] ?

Also, none of the 3 invariants regarding [[Extensible]]=false are
about
property names enumeration. Is there just no invariant to hold on
enumeration? If there is none, well, no need to do O(n)-ish
enforcements!

No thought on this point? That's pretty critical in terms of performance and could by itself compromise the consideration of the invariant enforcement model.

David

[1] esdiscuss/2011-March/013366