Save Object.observe()! (please) + make WeakMap/WeakSet observable.
Not sure I've got your idea right but I think the main point of WeakAnything is to forget about GC and have them not observable "at all costs" .... unless I've misunderstood the intent itself, Object.observe as it has been proposed wouldn't help much anyway.
On Mon, Nov 2, 2015 at 6:02 PM, Andrea Giammarchi < andrea.giammarchi at gmail.com> wrote:
Not sure I've got your idea right but I think the main point of WeakAnything is to forget about GC and have them not observable "at all costs"
That is true of WeakMap and WeakSet. But there is a separate WeakReference proposal in which GC would be observable.
The WeakReference proposal will once again be visible if wiki.ecmascript.org ever comes online again. What is the prognosis for that? There are many strong references into that wiki, so it should not have been collected.
2015-11-02 23:34 GMT+01:00 Coroutines <coroutines at gmail.com>:
I come from Lua. In Lua we make proxy objects with metamethods. You create an empty table/object and define a metatable with a __index and __newindex to catch accesses and changes when a key/property doesn't exist. I would primarily use this in sandboxes where I wanted to track the exact series of operations a user was performing to modify their environment (the one I'd stuck them in).
For this type of use case, you can use an ES6 Proxy < developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Proxy>.
You can think of the proxy handler's methods as the 'metamethods' of the proxy object.
What O.o would provide beyond Proxy is the ability to observe changes to already pre-existing objects. However, since you mention you'd start with an empty table/object, you should be able to create a fresh Proxy and use that to trace all property accesses.
Proxies are particularly well-suited when you want to sandbox things, since you should be in control of the sandboxed environment anyway and can set-up proxies to intermediate. O.o is particularly well-suited to scenarios where there are already plenty of pre-existing objects and you don't know ahead of time which ones to observe and which not.
I probably have a terrible understanding of how this all works at a low level but I feel like a potential solution would be a method of "upgrading" a non-proxy object to be a proxy. The reason accessors are being used as they are now is because you can retro fit them. Maybe what I am suggesting is essentially like swapping out the internal pointer of an object with another object (such as the way live module bindings work). In this way you might upgrade an existing object to behave like a proxy.
- Matthew Robb
That would make functional-programming-oriented developers wining forever about such monstrosity in specs ... I'd personally love such possibility!
Lol... I feel I'm in an insane minority that can work relatively productively in Java 7 and Haskell both.
Of course, I have a preference, but that preference lies around that of OCaml and Clojure. It's more the expression-based, impure functional languages that I'm most productive in. Observing mutations that I react to using immutable data structures. Sounds very odd and/or blasphemous to some, but that's what I like. MVC models like that are how Mithril and similar smaller frameworks have started to get some attention. It prefers highly local state, and an observed object would be a great state model for that.
And on that note, I'm going to stop before I derail the topic too far.
Isiah, could you elaborate some? I can't quite tell if you are expressing support for my suggestion or not.
I'm neutral.
Ah yes, the Smalltalk "become:" message, something that inspired SpiderMonkey's "Brain Transplants" -- see the [1] footnote at brendaneich.com/2010/11/proxy-inception, and of course the bugzilla link, which leads to this:
bugzilla.mozilla.org/user_profile?user_id=1214 Brendan Eich [:brendan] 2010-07-19 17:53:44 PDT
Burns: [saws off the top of Homer's head. No blood, very clean. The top of Homer's head rolls away.] Smithers, hand me that ice-cream scoop. Smithers: Ice-cream scoop?! Burns: Dammit, Smithers, this isn't rocket science, it's brain surgery!
("If I Only Had a Brain", from "Treehouse of Horror II")
So would I, it would open up some very efficient opportunities
For those interested in using Proxies in the future, here are the appropriate links to the two remaining holdouts (FF and MS Edge are already there):
Safari/Webkit/JSC:
bugs.webkit.org/show_bug.cgi?id=35731
Chrome/V8:
code.google.com/p/v8/issues/detail?id=1543
Bug the appropriate owners ;-).
On Tue, Nov 3, 2015 at 12:20 AM, Tom Van Cutsem <tomvc.be at gmail.com> wrote:
What O.o would provide beyond Proxy is the ability to observe changes to already pre-existing objects.
I think I missed something important here.
You can watch for events with both Proxy and Object.observe() - but Object.observe() lets you maintain the same object identity because you're not replacing references to the watched object.
Let's say I want to watch for events on an object I create: I create
a Proxy to that object and then expose references to that proxy (not
the target object). This is fine and dandy but if I want to observe
events on something like the String object/class (and its prototype)
I cannot replace all localized references to it in various modules.
Object.observe() could be seen as a 'cheat' in a sense that you can
observe events happening on the target object from code that is hidden
from you by a closure in another module (as long as you have a
reference to that object from where you use Object.observe()). With
Proxy you can only observe changes where you can replace the target
object's references.
With Object.observe() you get a global view of events generated by the target object, with Proxy you need to replace references to the target object with references to the Proxy.
Now I'm changing my opinion again. We need both.
But because of what Object.observe() can do to see into closures, I'd relegate it to privileged code :> It is incredibly useful for
debugging.
On Tue, Nov 3, 2015 at 3:22 PM, Coroutines <coroutines at gmail.com> wrote:
But because of what Object.observe() can do to see into closures, I'd relegate it to privileged code :> It is incredibly useful for debugging.
I really need to make sure my thoughts are complete before I send a message.
Anyway, I just wanted to say I'm viewing this through my experience with Lua.
In Lua you would make a proxy (like I mentioned before) by creating an empty table and setting the __index and __newindex metamethods to trigger when a key doesn't exist for an access or having its value set. This is the only way to watch for changes, you cannot attach a watch to an existing object - you create a separate object to act as the proxy and replace the references to the existing/target object. My last message was me realizing that Object.observe() has functionality that would let you see into encapsulated, hidden closures. This is why I think Object.observe() is cool but probably shouldn't be web-accessible.
Proxy is safe, Object.observe() should be debug-only functionality ((imo)). Like, in browsers you'd only have access to it from the console not from within a page?
There's a reason Object.observe is async: it prevents you from changing how the value is first assigned, so it can't work like a proxy. And question: how does it let you see hidden closures?
I was just thinking... Object.observe could, in theory, be used as a core for a dispatcher and store in a Flux-like data flow. Basically, when a view emits an event, you can use the first observed object as a memoized dispatcher, and then it emits a list of changes that can then update another observed object, the store. That store can then emit a list of changes to be resolved with the views.
What do you all think?
On Tue, Nov 3, 2015 at 7:33 PM, Isiah Meadows <isiahmeadows at gmail.com> wrote:
There's a reason Object.observe is async: it prevents you from changing how the value is first assigned, so it can't work like a proxy. And question: how does it let you see hidden closures?
In JS it is common to wrap everything in a (function (){})() for organization and/or encapsulation purposes. I would wrap a 'module' in that, and within that module I might define local references to common classes/libraries like: var string = require('String'); (example)
You cannot reach and redefine that private reference to String within that module, but you do still have a reference to String from /where you are/. If you Object.observe(String) from another place you can see events generated by the closed-over, private functions of that module.
In my opinion this is bad, as it breaks the expectation of privacy you have with a closure.
I think both should stay, but Object.observe() should not be accessible "from the web" and only used for debugging. With Proxy you can't watch events from existing references that are hidden behind closures. Proxy creates a separate, 2nd object so it does not have the same identity. If you want to make sure you catch all events on the target object you have to replace all the references to the target with the proxy. This is good, and does not allow closures to leak events from hidden references to the same target you're watching outside the closure.
You can't get anything related to actual object access from Object.observe. If you observe String, you can only get the following events:
- "add": String.newProp = value
- "update": String.existingProp = newValue
- "delete": delete String.prop
- "reconfigure": Object.defineProperty(String, ...)
- "setPrototype": String.proto = newProto
- "preventExtensions": Object.preventExtensions(String)
None of these can detect access. Only changes and removal.
And from each change, the callback is called with a list of objects with the following properties:
- "type": one of the types of actions above
- "object": the object being mutated
- "name": the name of the property
- "oldValue": if the action was "update", "delete", or "setPrototype", then the value it previously was. Otherwise, this property doesn't exist.
You can only get the old value from objects that existed before the closure. So it's already not necessarily unavailable outside the closure. You could achieve similar by replacing the original reference with a proxy before the closure, or even in theory by employing getters and setters. So it's not really breaking encapsulation beyond what's already possible in ES6. You can still leak closure references left and right with proxies. The difference is the fact you don't have to explicitly replace the reference beforehand. It could be implemented as an implicit reference replacement under the hood.
As for detecting object accesses, that remains impossible with Object.observe. The only thing you can do with it that you can't with proxies is to observe the global object itself, an uncommon use case. Everything else can be done with proxies. This just makes a common case easier.
As an aside and as Coroutines,
I never understood why there is this inability to enumerate WeakMap keys.
Back in the time of flash/as3, I made some modules to observe objects that were garbage collected, and this was extremely useful. This was only possible because the keys were utterable.
I was able to monitor objects at run time and their garbage collection as shown in this screenshot:
I also never really understood the statement there: tc39wiki.calculist.org/es6/weak-map
The AS3 tools (and demo for the curious): www.soundstep.com/blog/source/somacore/demo/Flash/SomaCoreGC
This would have a major impact in game development, I wish javascript would not limit itself constantly.
On Wed, Nov 4, 2015 at 4:56 AM, Romuald Quantin <romu at soundstep.com> wrote:
As an aside and as Coroutines,
I never understood why there is this inability to enumerate WeakMap keys.
If I had it my way there would be no WeakMap or WeakSet. I'd have a Symbol.mode similar to Lua's __mode meta(method/field?) that would show the garbage collector that keys or values are weakly referenced in an object. Or even:
obj[Symbol.weakValues] = true; obj[Symbol.weakProperties] = true;
WeakMap isn't really all that special. Set is important for how it maintains only unique values are inserted - but you could do this with Proxy. I think I come across as a Lua fanboy... I enjoyed the minimalism of Lua for sure :-)
I wish I could have weak references in arrays... :3 Iterate to .length, skipping undefined/null values (where things have been collected).
2015-11-03 15:41 GMT+01:00 Matthew Robb <matthewwrobb at gmail.com>:
I probably have a terrible understanding of how this all works at a low level but I feel like a potential solution would be a method of "upgrading" a non-proxy object to be a proxy. The reason accessors are being used as they are now is because you can retro fit them. Maybe what I am suggesting is essentially like swapping out the internal pointer of an object with another object (such as the way live module bindings work). In this way you might upgrade an existing object to behave like a proxy.
This feature was proposed (under the name Proxy.startTrapping) back in 2011-2012. See < web.archive.org/web/20140426153405/http://wiki.ecmascript.org/doku.php?id=strawman:direct_proxies#proxy.starttrapping_a.k.a._proxy.attach
.
The two arguments that killed it, IIRC:
-
If a module A hands out a reference to, say, a function f to modules B and C, then C could use this primitive to replace f with its own proxied version. Module B expects f to work as A intended, but module C can completely override its behavior, stealing any arguments to the function that B would pass. This is really bad behavior from a security and modularity perspective.
-
As Brendan mentioned, implementing this primitive requires a Smalltalk-like "become:" that can swap object identities. This can be really tricky or costly to implement (highly dependent on how objects are represented in the VM).
2015-11-04 0:22 GMT+01:00 Coroutines <coroutines at gmail.com>:
With Object.observe() you get a global view of events generated by the target object, with Proxy you need to replace references to the target object with references to the Proxy.
Now I'm changing my opinion again. We need both.
I won't speak to whether we really need both, but your analysis that they cater to different use cases is correct:
-
O.o allows one to asynchronously observe (not intercept) operations on another one's objects. This is safe.
-
Proxies allow one to synchronously intercept operations on one's own objects. This is safe.
-
Proxies can obviously also be used to just observe one's own objects. As an existence proof, I once did a simple O.o polyfill using proxies < tvcutsem/harmony-reflect/blob/master/examples/observer.js>.
This is not generally useful though.
- Proxy.startTrapping would allow one to synchronously intercept operations on another one's objects. This way lies madness. Don't go there.
On Wed, Nov 4, 2015 at 4:46 PM, Tom Van Cutsem <tomvc.be at gmail.com> wrote:
- If a module A hands out a reference to, say, a function f to modules B and C, then C could use this primitive to replace f with its own proxied version. Module B expects f to work as A intended, but module C can completely override its behavior, stealing any arguments to the function that B would pass. This is really bad behavior from a security and modularity perspective.
It seems like a straight forward solution for this might be adding
something like Proxy.preventTrapping(...) and have this applied to all
module exports/imports by default. Since modules work off bindings and not
object properties.
- Matthew Robb
On Wed, Nov 4, 2015 at 8:16 AM Coroutines <coroutines at gmail.com> wrote:
On Wed, Nov 4, 2015 at 4:56 AM, Romuald Quantin <romu at soundstep.com> wrote:
As an aside and as Coroutines,
I never understood why there is this inability to enumerate WeakMap keys.
If I had it my way there would be no WeakMap or WeakSet. I'd have a Symbol.mode similar to Lua's __mode meta(method/field?) that would show the garbage collector that keys or values are weakly referenced in an object. Or even:
obj[Symbol.weakValues] = true; obj[Symbol.weakProperties] = true;
WeakMap isn't really all that special. Set is important for how it maintains only unique values are inserted - but you could do this with Proxy.
This is all very misunderstood (both Romuald and Coroutines comments). WeakMap is extremely special, for these reasons:
- The weakly mapped relationship between the key and value allows the runtime to gc the value once the key becomes otherwise unreachable. Specifically: if the only reachable reference to the key is the weakmap relationship, then that relationship can be discarded and the value can be garbage collected at some implementation-determined time in the future.
- For security purposes, only code that has been explicitly given access to both the weakmap and a key may gain access to the value stored within the weakmap. This is why there is no enumeration—if there were, then any code that could access the weakmap could then also access all of the keys and values. If that's a desirable trait, then just use a Map.
These things cannot be done with Proxy, because anything that Proxy does will inevitably be a strongly held reference.
On Wed, Nov 4, 2015 at 2:36 PM, Rick Waldron <waldron.rick at gmail.com> wrote:
On Wed, Nov 4, 2015 at 8:16 AM Coroutines <coroutines at gmail.com> wrote:
obj[Symbol.weakValues] = true; obj[Symbol.weakProperties] = true;
- For security purposes, only code that has been explicitly given access to both the weakmap and a key may gain access to the value stored within the weakmap. This is why there is no enumeration—if there were, then any code that could access the weakmap could then also access all of the keys and values. If that's a desirable trait, then just use a Map.
These things cannot be done with Proxy, because anything that Proxy does will inevitably be a strongly held reference.
Well, you explained the mystery behind why WeakMap can't be iterated over - but I was saying I would combine Proxy with a theoretical Symbol for marking keys and/or values as weakly-referenced to create WeakSet and WeakMap (if I wanted). I think I just prefer Lua in this area:
local tmp = setmetatable({ 'a', 'b', 'c' }, { __mode = 'v' })
for k, v in ipairs(tmp) do print(k, v) end
'a', 'b', and 'c' are weakly-referenced and can be iterated over, will be marked by the next GC step, and will not survive the next GC collection cycle. I wish JS let me create weak references as simply:
var tmp = [ 'a', 'b', 'c' ]
tmp[Symbol.weakValues] = true;
This is what I was saying :> I understand that you need both a
reference to the WeakMap and the object you're using as a "weak key" but I think it could be iterable if the weak references have not been broken yet - if the objects in that WeakMap have not been collected yet. I would not want "permanently" strong references with a Map.
- For security purposes, only code that has been explicitly given access
to both the weak map and a key may gain access to the value stored within the > weakmap. This is why there is no enumeration—if there were, then any
code that could access the weak map could then also access all of the keys and > > values. If that’s a desirable trait, then just use a Map.
What is the problem in accessing the keys with any code, which is the same behaviour for any other object in javascript: String, Array, Object and so on. The point is being able to do on a weak key so code can be processed without being retained by the GC.
Someone made an experiment with the Mozilla “getWeakReference” apparently.
gist.github.com/Benvie/7123690, developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_Bindings/Components.utils.getWeakReference
I had this as granted with other languages, I still don’t understand why has this not been enabled by default in javascript.
Read esdiscuss.org, use Google site: search, before complaining that you don't understand why something is not "enabled". Here's a link:
It's not hard to find the full history of the discussion.
On Thu, Nov 5, 2015 at 12:30 AM Romuald Quantin <romu at soundstep.com> wrote:
What is the problem in accessing the keys with any code, which is the same behaviour for any other object in javascript: String, Array, Object and so on. The point is being able to do on a weak key so code can be processed without being retained by the GC.
That's one point, but not the only one. Think of a WeakMap wm as the dual of a private field pf, named by a symbol:
obj[pf] // get the value of a symbol-named field wm[obj] // get the value of a weak-mapped field
A second point of not enumerating weakmap keys is to enable engines to transpose from the wm[obj] to the obj[pf] form, unobservably. I believe the Chakra engine in IE and Edge does this. It's a performance win, as usually the key (obj) outlives -- or lives only as long as -- the map (wm), so hanging the map as a pf off the obj's hidden class causes no memory leak due to an unwanted strong reference.
The untransposed case requires the JS engine's garbage collector to treat weakmaps specially, with bad worst-case complexity (think of tying knots among keys and values in a weakmap or set of maps -- a global mark/sweep phase would be required). More on weakmap GC overhead: esdiscuss.org/topic/linear-time-weak-map-gc (clever improvement there, but still hairy compared to the transposed case, which is just objects with hidden classes, etc., in all engines today, plus private field keys that can't be enumerated).
A third point is the one Rick cited: in an Object Capability security model, giving a capability to wm should not give you all the keys. This is the POLA in action. Yes, someone debugging a system built with weakmaps would want the keys. In a stratified host/guest scenario, the host wants to debug the guest's code and heap, including finding all the keys. But for the guest there should be no capability leakage.
2015-11-04 23:04 GMT+01:00 Matthew Robb <matthewwrobb at gmail.com>:
On Wed, Nov 4, 2015 at 4:46 PM, Tom Van Cutsem <tomvc.be at gmail.com> wrote:
- If a module A hands out a reference to, say, a function f to modules B and C, then C could use this primitive to replace f with its own proxied version. Module B expects f to work as A intended, but module C can completely override its behavior, stealing any arguments to the function that B would pass. This is really bad behavior from a security and modularity perspective.
It seems like a straight forward solution for this might be adding something like
Proxy.preventTrapping(...)and have this applied to all module exports/imports by default. Since modules work off bindings and not object properties.
We've thought about preventing certain objects from becoming trapped. However, my explanation above only used modules to frame the discussion. The security/modularity problem is in no way tied to module export/import bindings only. Code can be modularised in ways other than using modules, the most common one being multiple <script> tags on a page, good old
CommonJS modules, etc. In other words: for most JS code (and virtually all legacy code), module boundaries are implicit. Even if they were explicit, if two modules start to exchange object references, you transitively don't want to have those objects be trapped either, so just defending module boundaries would not even be sufficient.
In addition, limiting the scope of Proxy.startTrapping to fewer objects does not address the second point about implementation complexity.
O.o is a much safer alternative to Proxy.startTrapping and covers at least part of its use cases.
On Thu, Nov 5, 2015 at 12:49 PM, Tom Van Cutsem <tomvc.be at gmail.com> wrote:
O.o is a much safer alternative to Proxy.startTrapping and covers at least part of its use cases.
Hmm.
Okay so pretend all object's are really just proxies to whatever their
internal backing might be. In this scenario perhaps what we might want is
Proxy.observeTraps which should work just like Object.observe only
synchronously and without a baked in ability to intercede. One could,
however, respond to a set operation by resetting the old value. What this
does is give you something VERY close to a built in observable Model type
on top of plain objects.
- Matthew Robb
Proxy at the granularity of a property would be perfect for that
You wrote
-
"pretend all object's are really just proxies to whatever their internal backing might be."
-
"without a baked in ability to intercede."
-
"What this does is give you something VERY close to a built in observable Model type on top of plain objects."
These are not consistent statements. Either plain objects with nothing on top or underneath support synchronous intercession, or they do not. TC39 for reasons Tom Van Cutsem listed has decided "not". Implementors agree.
I did watchpoints in SpiderMonkey in 1997 IIRC, implemented as replacement accessors under the hood. ES5 and unfrozen ordinary objects allow self-hosting of watchpoints (ES6 weakmaps and symbols help but are not required).
We can't standardize watchpoints for all objects, of course.
I was referred to es-discuss after filing a feature request here to support observing WeakMaps and WeakSets. ( bugzilla.mozilla.org/show_bug.cgi?id=1206584 )
I was linked this message: esdiscuss/2015-November/044684
Please save Object.observe().
When I read that above posting it sounds like O.o() is to be removed because it has not been popularly used. It is proposed for ES7 - it's not even part of an accepted standard yet. I think it's strange to remove something because it's not used much... because it's experimental. Who would depend on something non-standard? Sure there's a polyfill but it polls an object for changes with timers - this sounds ridiculously inefficient. If it doesn't get accepted there will be no way to watch changes as they happen, instead of after the fact.
I come from Lua. In Lua we make proxy objects with metamethods. You create an empty table/object and define a metatable with a __index and __newindex to catch accesses and changes when a key/property doesn't exist. I would primarily use this in sandboxes where I wanted to track the exact series of operations a user was performing to modify their environment (the one I'd stuck them in). The important part to remember is that I didn't know what properties of which objects they would be accessing. I couldn't just define a static list of getters and setters and hope they make use of my limited, catching interface. I think Object.observe() is very important for situations where you want to watch all changes, because you don't know which properties will be used.
I have a further proposal: I wish it were possible to watch changes on WeakMaps and WeakSets.
In Lua you can have weak references by setting the __mode = 'kv' (k for key, v for value) in the object's metatable. The only way I could get weak references in JS was with WeakMaps and WeakSets. Lua also lets you set a __gc in the metatable to handle when a pair is collected by the GC. I wanted the ability to "see" when memoized function calls are forgotten, but I can't do that with just Object.observe() - as you can't observe a WeakSet or WeakMap. I'd love for this to be possible.
For debugging reasons it would be excellent to be able to watch when things get collected (through weak references). Finalizers in general would be nice. I mean in the sense that I want to do something independent of the object being collected, not that I need to destroy the object in a special way before collection. I feel like there's a vague difference between finalizer and destructor.
Please save Object.observe(), make WeakMap/WeakSet observable, and save the whales too or something.
Toodles ~