Object.observe and observing "computed properties"
Thanks for this analysis. I globally agree with your reasoning, but I don't think this is much of a problem.
Certainly, dependency analysis isn't completely effective, in the sense that each time you get 'out' of your 'observable world', you may miss updates. This is, however, not such a big issue as most binding languages feature that kind of 'issues', even your 'observe' API. This is just a fact that 'Math.random()' and 'window.innerWidth' can't be observed, and it's up to the developer to make sure it they are not in their code, just like it's your reponsability to notify for accessors if you're using 'Object.observe'.
Most of the times, when you're binding your View to you Model, your model only lives in a particular world where it only interacts with other model objects, so that you can control completely the model world to make it observable, and you shouldn't have 'lost' dependency issues.
However, when there's no other possibilities (but it's really rare) than to use 'unobservable' properties, the only solution left to you is to use an event-based approach ('window.innerWidth + window.onresize') or perform polling ('Math.random() + setInterval'). My own JS library has something similar called 'binding bridges' which basically maps the properties of an object to its dynamic value, and updates them has necessary using a custom model (event or setInterval) :
var BindingBridges = Object.makeBindable({});
BindingBridges.add = function(name,evaluator,timespan) {
BindableObject.methods.addProperty.call(this,name);
BindingBridges[name] = evaluator();
setInterval(function() {
var cValue = evaluator();
if (BindingBridges[name]==cValue) { return; }
else { BindingBridges[name]=cValue; }
}, timespan);
}
// you could also use an event-driven approach and
// set BindingBridges.property to its new value at
// each call of the event, instead.
Developers relying on unobservable things can use such mechanism to solve their issue. For example, in my 'mail' template application, the time was displayed as 'seconds ago', 'less than 1 minute ago', '? minutes ago', '? hours ago', ... and was relying on a 'BindingBridge.now' property which was updated using setInterval every 20 seconds or so.
As for the observation of an array, the classical 'dependency' pattern is sufficient, most of the times, since you'll depends on the 'array.length' property of your array (when an item is inserted or deleted, you'll be notified) and on the 'array[i]' value you touched.
var o = {
values: [0,1,5,-3],
get max() {
var max = Number.negativeInfinity;
this.values.forEach(
(i) => (i>max && max=i)
);
return max;
}
}
Object.makeBindable(o);
new Binding(=> o.max, v => $("#min").textContent=v);
o.values.push(-10); // works
o.value[o.values.length-1] = 3; // works
However, listening to a collection is something different from listening to a simple object. My own JS library used a BindableArray class which, indeed, generated 'insert' and 'delete' events in addition to the tradtionnal 'update' events, as well as a HTMLForEachBinding which listened specifically to array events and managed the DOM as a close copy of the array using a binding template; those 'insert' and 'delete' events could be transferred to a 'made-observable' arrays.
-----Message d'origine---
Hey
Thanks very much for your detailed response! I totally agree that the question can be distilled down to "deciding when [a] function should be re-evaluated", and that two approaches are "2a: requiring function author to declare [dependencies]" or "2b: attempt to discover [dependencies]".
To clarify why Knockout/Batman/CanJS/etc developers make use of dependency detection (and why for them it's necessary for it to be integrated into observability), there are two main scenarios:
[1] Computed properties whose output depends on observable state. Example:
- this.grandTotal = function() {*
-
var result = 0;* -
for (var i = 0; i < this.items.length; i++) { result +=
this.items[i].price; }*
-
return result;* - };*
Knockout developers are used to this sort of thing updating automatically whenever you modify the price of any item, or when you add or remove items to the array. It would be very inconvenient to have to somehow declare dependencies manually (2a) - I'm not even sure what kind of syntax or mechanism you could use when the set of dependencies changes over time. That leaves option (2b) which works great, as long as dependency detection is built into observability.
As Rafael correctly points out, some functions may return nondeterministic values, or values that continually change even given the same inputs (e.g., new array instances). In practice that doesn't cause any downside in real KO applications - you just receive the latest return value and work with it. It's true that evaluator functions that mutate their own dependencies are potentially bad (in the worst case, stack overflow), but in practice no worse than regular recursive functions.
[2] Functions whose effect varies based on observable state. Example:
- function displayPersonInfo() {*
-
$(".name").text(myModel.name.first);* -
$(".age").text(myModel.age);* -
$(".isTall").val(descriptions.isPersonTall(myModel.id));* -
console.log("Refreshed display for person " + myModel.id);* - }*
- someLibrary.autoRefresh(displayPersonInfo);*
The facility here is being able to take action when one of a set of things change. This is a core part of why it's easy to write custom binding logic in KO - you simply provide a callback that accesses arbitrary data and does something (without having to declare dependencies or subscribe/unsubscribe to anything), and you know it will be re-run when necesssary.
Overall, I've no wish to derail the Object.observe proposal, and fully accept that dependency detection might end up being out of scope for it. However if the ES recommendation will end up being "solve this problem through convention, not language support", I'd love to have a sense of what kinds of conventions (examples, preferably) we would actually be recommending and how they would offer a similar level of convenience to dependency detection.
François's proposal of a "make bindable" function is certainly interesting, and I wonder to what extent it could tie in with Object.observe. Perhaps you could have some function that meant "run this callback, and use Object.observe to set up a subscription to anything bindable that the callback touches". If so that would have pretty much exactly the right semantics for the scenarios I'm considering, and the fact that objects aren't bindable by default would mitigate perf costs.
Thanks again to Rafael for leading the charge on this and making web development better in the long run!
Steve
On Wed, Aug 29, 2012 at 11:09 AM, Steve Sanderson <flares at gmail.com> wrote:
Hey
Thanks very much for your detailed response! I totally agree that the question can be distilled down to "deciding when [a] function should be re-evaluated", and that two approaches are "2a: requiring function author to declare [dependencies]" or "2b: attempt to discover [dependencies]".
To clarify why Knockout/Batman/CanJS/etc developers make use of dependency detection (and why for them it's necessary for it to be integrated into observability), there are two main scenarios:
[1] Computed properties whose output depends on observable state. Example:
- this.grandTotal = function() {*
var result = 0;*for (var i = 0; i < this.items.length; i++) { result +=this.items[i].price; }*
return result;*- };*
Knockout developers are used to this sort of thing updating automatically whenever you modify the price of any item, or when you add or remove items to the array. It would be very inconvenient to have to somehow declare dependencies manually (2a) - I'm not even sure what kind of syntax or mechanism you could use when the set of dependencies changes over time. That leaves option (2b) which works great, as long as dependency detection is built into observability.
I'm not sure that's true. Side-effects are a real pain and it seems to me that there's going to be some practical advice at the bottom of any of these systems that says, in effect, "don't do things we can't understand". That sort of advice is likely to be backed up with tools to assist you in helping developers understand those limits; say transpiler passes that analyze the dependencies in a function.
It strikes me that this is at some level a question of how deep your analysis of the target function is willing to go. So far the examples dependencies are only on in-scope objects inside a computed property's generator. But what about methods called there that might have inputs that change? How deep does the propagation go?
It seems that the implicitness of this strategy implies that some computed properties will be always marked "regenerate" as it'll be simpler/easier/faster than doing something more sophisticated.
As Rafael correctly points out, some functions may return nondeterministic
values, or values that continually change even given the same inputs (e.g., new array instances). In practice that doesn't cause any downside in real KO applications - you just receive the latest return value and work with it. It's true that evaluator functions that mutate their own dependencies are potentially bad (in the worst case, stack overflow), but in practice no worse than regular recursive functions.
[2] Functions whose effect varies based on observable state. Example:
- function displayPersonInfo() {*
$(".name").text(myModel.name.first);*$(".age").text(myModel.age);*$(".isTall").val(descriptions.isPersonTall(myModel.id));*console.log("Refreshed display for person " + myModel.id);*- }*
- someLibrary.autoRefresh(displayPersonInfo);*
The facility here is being able to take action when one of a set of things change. This is a core part of why it's easy to write custom binding logic in KO - you simply provide a callback that accesses arbitrary data and does something (without having to declare dependencies or subscribe/unsubscribe to anything), and you know it will be re-run when necesssary.
Overall, I've no wish to derail the Object.observe proposal, and fully accept that dependency detection might end up being out of scope for it. However if the ES recommendation will end up being "solve this problem through convention, not language support", I'd love to have a sense of what kinds of conventions (examples, preferably) we would actually be recommending and how they would offer a similar level of convenience to dependency detection.
I don't think they will, frankly. The best of them will re-create dependency detection via compiler. The less aggressive may simply force enumeration of dependencies or create conventions which cause particular properties to be observed through participation.
From: Alex Russell Sent: Thursday, August 30, 2012 7:44 PM To: steven at stevensanderson.com Cc: es-discuss at mozilla.org Subject: Re: Object.observe and observing "computed properties"
On Wed, Aug 29, 2012 at 11:09 AM, Steve Sanderson <flares at gmail.com> wrote:
Knockout developers are used to this sort of thing updating automatically whenever you modify the price of any item, or when you add or remove items to the array. It would be very inconvenient to have to somehow declare dependencies manually (2a) - I'm not even sure what kind of syntax or mechanism you could use when the set of dependencies changes over time. That leaves option (2b) which works great, as long as dependency detection is built into observability.
I'm not sure that's true. Side-effects are a real pain and it seems to me that there's going to be some practical advice at the bottom of any of these systems that says, in effect, "don't do things we can't understand". That sort of advice is likely to be backed up with tools to assist you in helping developers understand those limits; say transpiler passes that analyze the dependencies in a function.
Using a transpiler to detect the dependencies would be very difficult; avoiding memory leaks seems nearly impossible in this case. KnockoutJS features automatic dependency detection for years and I don’t think it has raised any issue at this time. Developers do not bind an UI element to a function that actually does something else than formatting a value or doing a computation (ie: readonly methods). I think it would be safe to say that, in “dependency tracking mode” the observable objects are read-only (you can’t modify them or it throws) so that it’s impossible to use ill-suited methods as a source of binding.
Also, to continue on your ‘transpiler’ idea: how would a transpiler work to detect changes to the dependency properties? Would you require JS code to receive every read and write notifications for all properties observable objects (like it’s the case in Object.observe), filter them to find the interesting bits, and mark themselves the modified properties and the bindings which are not up-to-date anymore?
In such case, a proxy polyfilling the API I propose will be way smarter...
It strikes me that this is at some level a question of how deep your analysis of the target function is willing to go. So far the examples dependencies are only on in-scope objects inside a computed property's generator. But what about methods called there that might have inputs that change? How deep does the propagation go?
If the arguments of the method “touch” observable objects, they will be watched for modifications, just like any other dependency. If they are static, they will not fire traps in observable objects and will not cause overhead.
It seems that the implicitness of this strategy implies that some computed properties will be always marked "regenerate" as it'll be simpler/easier/faster than doing something more sophisticated.
Could you develop? I don’t get that issue.
Overall, I've no wish to derail the Object.observe proposal, and fully accept that dependency detection might end up being out of scope for it. However if the ES recommendation will end up being "solve this problem through convention, not language support", I'd love to have a sense of what kinds of conventions (examples, preferably) we would actually be recommending and how they would offer a similar level of convenience to dependency detection.
I don't think they will, frankly. The best of them will re-create dependency detection via compiler. The less aggressive may simply force enumeration of dependencies or create conventions which cause particular properties to be observed through participation.
Avoiding memory leaks using a compile-time dependency tracking seems a nightmare to me, and detecting affected bindings would be a bummer. Bad idea flag raised.
Following this discussion, Rafael and I talked about various strategies that an MV* library could use to detect dependencies from arbitrary expressions and code blocks, as would be needed to achieve the kinds of niceties present in Knockout.js/Batman.js etc. Some of these tie in with Object.observe more than others.
The upshot is that we don't have a single most compelling way to do this kind of dependency detection with Object.observe, but it would be possible to augment Object.observe to add that ability in a future language version (see technique A below), or alternatively, libraries could use somewhat less clean techniques to achieve it with proxies/accessors alone, regardless of Object.observe (see techniques B and C below).
So as a library developer I would in principle be happy to see Object.observe added to ES, since it appears to be a step in the right direction. But I would caution that unless read notifications were added (see technique A), my library (knockout.js) couldn't use it to achieve something as clean as its existing semantics, so we'd most likely be waiting for future improvements before being able to use Object.observe.
Steve
Appendix: the techniques we considered
- Technique A: Add read notifications to Object.observe (either now, or
in a future ECMAScript version)
- Just as the existing Object.observe proposal delivers notification of property writes for specific objects, a symmetrical API could deliver notification of property reads for specific objects
- Pros: it requires no proxies or accessors, so works with raw data
- Cons: requires language support, like Object.observe does for write notifications
- Technique B: Membrane-style proxies transitively capture all chains
of properties read from a given object
- The logic that parses/evaluates binding expressions could supply a specially wrapped version of the underlying data that uses a proxy to log property reads
- Pros: doesn't require language support; ties in with Object.observe in that once you know what properties were read, you can use Object.observe to subscribe to write notifications
- Cons: because the proxies aren't the real underlying data objects, it's possible to get into confusing scenarios where things don't work, for example if you read a property of a closure-captured variable, that won't be logged and hence the framework can't auto-update when that property changes
- Technique C: Monkeypatch all model data properties with accessors
that log reads
- Some utility function would walk the object graph and replace all properties with special accessors
- Pros: doesn't require language support; also doesn't even require Object.observe since you might as well also replace setters with ones that trigger notification on change
- Cons: very intrusive - permanently modifies the developer's data structures
From A, B and C as alternatives, especially A and C, it seems you want synchronous read intercession, but are happy with Object.observe's asynchronous write intercession?
If so, then the plan interference objection is fatal.
If not, then how would async read intercession work? Callbacks and promises are not going to fly.
[Sorry, this time from the right email]
Object.deliverChangeRecords allows A to be a strategy in the possible future that Steve describes (if reads were to generate changeRecords)
[Just to be clear: I don't think it makes sense to include reads in the set of changeRecords generated by Object.observe() at this point]
The basic idea is that every "computed property" has "around advice" that clears its pending delivery queue before and after invocation of the actual function so it can disambiguate reads generated during the invocation from outer (previous or later) or inner (sub-invocations of computed properties) reads.
Rafael Weinstein wrote:
[Just to be clear: Idon't think it makes sense to include reads in the set of changeRecords generated by Object.observe() at this point]
Indeed -- one might want to rename these things from "changeRecords" to something else, at least!
But beyond the name, reads dominate writes. That raises the overhead, and also makes me wonder about the ability of an observer to make sense of what's going on, given that all these records must be processed in a later turn. Sure, the records will be well-ordered, but at that late date, do the reads matter? The reader (getter) got whatever data it wanted, synchronously, get by get.
Steve Sanderson (author of the excellent KnockoutJS framework) and François REMY both raised the issue of observing computed properties WRT to the Object.observe() proposal.
Having thought about this problem while helping to author Object.observe, my thoughts are as follows:
First, I think you need to look at the problem the right way. "Observing computed properties" is kind of non-sensical. What this translates to is "observe when the return value of an anonymous function invocation will be different from the last invocation".
I think the only reasonable way to look at it is this: imagine that a data property can be defined whose value can only be set by assigning it the return value of a given function. Observing when a data property changes value is trivial. The problem becomes: deciding when the function should be re-evaluated.
Looked at like this, the first thing that becomes obvious is that there are a class of functions which should never be used: those functions that can "spontaneously" change value. e.g.
var i = 1; function getVal { return ++i; } function getVal2 { return Math.random(); } function getVal3 { return myElement.offsetHeight; }
[I actually think at this point it becomes clear that, because the task at hand isn't solvable in general, it's not appropriate to include support for it at the language level, but lets continue].
That said, those functions:
-whose output is dependent only on the value of a discrete set of inputs, (i.e. are stateless) -don't modify their inputs -will always return the same output value for the same set of inputs
can be sensibly used for computed property functions. It's worth noting that even this can be easy to get wrong. For example, many webdevs might not realize that
var firstName = 'Rafael'; var lastName = 'Weinstein'; function getVal() { return [firstName, lastName]; }
doesn't meet this criteria as its return value is always a different Array object.
Assuming that you've been careful to pick an appropriate function, there are two approaches to knowing when to reevaluate it:
At this point, you have only trade-offs. (1) Is potentially expensive and hard to do at exactly the right time, but (2) requires having proper knowledge of the function's inputs.
Obtaining the set of inputs for (2) can be done in two ways:
2a) Require the function author to declare them. 2b) Attempt to discover them by running the function and observing which inputs it accesses.
(2a) requires some careful attention on the part of the function author, so in some sense, if (2b) were possible, it would be ideal. This brings us to what KnockoutJS does and what François proposed, so let's consider it.
The first problem is what I discussed above, that creating an appropriate function is potentially tricky and/or hard to understand, and there isn't any way to statically determine if a function is or is not appropriate.
The second problem is that doing this risks "discovering" inputs that aren't really inputs at all. In other words, the function, just be being invoked happens to "touch" a wide swath of objects, even though they aren't dependencies of the function. This is bad because it would cause the system to "observe" these objects, which, given modern VMs, will cause them to "de-optimize" and become slower to access
Thus, offering language-level support for (2b) puts developers in the risky situations of authoring computed property functions which may
A) not fire their notifications, even though they appear to have "changed value" B) become a "go-slow" button for their entire application
...with no good recourse to discover why either is happening.
Note that François's proposal included a mitigation of (B), in that you need to whitelist objects as potential discoverable dependencies. This helps some with the risk of "discovering too many" dependencies, but it also risks "not discovering enough" dependencies, which becomes problem (A) again.
Thus, what we have is a problem which is really solved through convention, not through an abstract solution, and thus the most sensible thing to do is leave it to authors to make trade-offs for themselves about the pros & cons of the various approaches and the conventions they imply.
Steve Sanderson (author of the excellent KnockoutJS framework) and François REMY both raised the issue of observing computed properties WRT to the Object.observe() proposal. Having thought about this problem while helping to author Object.observe, my thoughts are as follows: First, I think you need to look at the problem the right way. "Observing computed properties" is kind of non-sensical. What this translates to is "observe when the *return value* of an anonymous function invocation *will be* different from the last invocation". I think the only reasonable way to look at it is this: imagine that a *data property* can be defined whose value can only be set by assigning it the return value of a given function. Observing when a data property changes value is trivial. The problem becomes: deciding when the function should be re-evaluated. Looked at like this, the first thing that becomes obvious is that there are a class of functions which should never be used: those functions that can "spontaneously" change value. e.g. var i = 1; function getVal { return ++i; } function getVal2 { return Math.random(); } function getVal3 { return myElement.offsetHeight; } [I actually think at this point it becomes clear that, because the task at hand isn't solvable in general, it's not appropriate to include support for it at the language level, but lets continue]. That said, those functions: -whose output is dependent only on the value of a discrete set of inputs, (i.e. are stateless) -don't modify their inputs -will always return the same output value for the same set of inputs can be sensibly used for computed property functions. It's worth noting that even this can be easy to get wrong. For example, many webdevs might not realize that var firstName = 'Rafael'; var lastName = 'Weinstein'; function getVal() { return [firstName, lastName]; } doesn't meet this criteria as its return value is always a different Array object. Assuming that you've been careful to pick an appropriate function, there are two approaches to knowing when to reevaluate it: 1) Dirty-checking: Immediately before each time you "need" the value of the dependent property. 2) Dependency observation: When one or more of its inputs have changed value. At this point, you have only trade-offs. (1) Is potentially expensive and hard to do at exactly the right time, but (2) requires having proper knowledge of the function's inputs. Obtaining the set of inputs for (2) can be done in two ways: 2a) Require the function author to declare them. 2b) Attempt to discover them by running the function and observing which inputs it accesses. (2a) requires some careful attention on the part of the function author, so in some sense, if (2b) were possible, it would be ideal. This brings us to what KnockoutJS does and what François proposed, so let's consider it. The first problem is what I discussed above, that creating an *appropriate* function is potentially tricky and/or hard to understand, and there isn't any way to statically determine if a function is or is not appropriate. The second problem is that doing this risks "discovering" inputs that aren't really inputs at all. In other words, the function, just be being invoked happens to "touch" a wide swath of objects, even though they aren't dependencies of the function. This is bad because it would cause the system to "observe" these objects, which, given modern VMs, will cause them to "de-optimize" and become slower to access Thus, offering language-level support for (2b) puts developers in the risky situations of authoring computed property functions which may A) not fire their notifications, even though they appear to have "changed value" B) become a "go-slow" button for their entire application ...with no good recourse to discover why either is happening. Note that François's proposal included a mitigation of (B), in that you need to whitelist objects as potential discoverable dependencies. This helps some with the risk of "discovering too many" dependencies, but it also risks "not discovering enough" dependencies, which becomes problem (A) again. ---- Thus, what we have is a problem which is really solved through convention, not through an abstract solution, and thus the most sensible thing to do is leave it to authors to make trade-offs for themselves about the pros & cons of the various approaches and the conventions they imply.