Improving ECMAScript as a compilation target
On Feb 12, 2009, at 8:07 PM, Peter Michaux wrote:
About five months ago I posted here about improving ECMAScript as a compilation target.
esdiscuss/2008-September/007652
I was disappointed that there weren't any responses to that email. Folks were probably busy with other concerns. I also thought there might be some motivation to avoid making ECMAScript an improved compilation target as that may reduce the number of programmers working directly in ECMAScript. Perhaps that goes against the goals of this group.
That does not make sense. It assumes there is some control by the
committee on the number of programmers hand-coding ECMAScript. There
isn't. Rather, it's extremely unlikely that hand-coding JS will cease
any time soon, no matter the increase in code generators.
Therefore the language should not have footgun/power-tools of the kind
a compiler writer would want in a memory-safe compilation target
language (e.g. call/cc). A while ago when this topic came up (well
before your last post on the subject), in esdiscuss/2006-June/003482
, Lars wrote:
I don't think that being a compiler target is among the core use cases for ECMAScript 4. (Nor do I think it should be.) I also don't think that a pragmatic programming language like ECMAScript should follow Scheme, say, in providing a minimalist and powerful core on which a great variation of abstractions can be written by the initiated. Instead I think ECMAScript needs to be a rich language with features that are immediately useful to its core audiences, and which play well with each other in the sense that they have unsurprising consequences when composed.
If you s/ECMAScript 4/ECMAScript Harmony/ I still agree completely,
and I expect others on the committee do too. We're still focused on
programming in the large, higher integrity, lexical scope, and
appropriate improvements to usability and expressiveness for a large
number of programmers -- not optimization for automatic code generators.
As for trying to be both a desert topping and a floor wax, that trick
never works in reality.
Certainly we can make things like ES3.1 strict mode better for certain
code generators such as Caja and AdSafe, but by adding integrity
property support. This helps hand coders too, we believe -- even if we
do not expect "use strict" to predominate any more than code
generation to take over in the near term.
The general browser scripting world is still programming in ECMAScript; however, the number of projects that compile to ECMAScript are increasing. I expect to see this trend continue to grow.
Who knows what the balance will be, but again: hand-coding will
predominate for the near term and it's unlikely to drop to an
insignificant fraction of the JS on the web.
Note that more than a few code generators take a source language
related to JS (subset, close cousin, slightly extended dialect). This
is not a coincidence; it eases the pain of inevitable abstraction
leaks and helps debugging. It matches impedances, reducing risk of
semantic conflicts and performance surprises.
I wrote the following as a blog post recently. I don't presume anyone here reads my blog so am submitting it here. I am interested to know if attitudes have changed or people are warming up to the idea of changing ECMAScript intentionally to make it a better compilation target.
If I wanted a safe intermediate form, I would look at arithmetic
coding techniques that self-verify (so that you can't express other
than well-formed programs), not bytecode and not JS source. It's IMHO
very unlikely that TC39 will standardize a bytecode for JS. An AST
encoding has better chances and better properties including self-
verification and view-source-ry.
The same is happening in the browser. Developers are compiling many languages down to ECMAScript ("they byte code of the browser"). GWT and Cappuccino are two well know examples. These projects are overruling the idea that they must write in ECMAScript to create browser apps. They don't have to wait for the ECMAScript committee to standardize methodMissing, for browsers to implement it and for old browsers to disappear. Ten years? No way! They have methodMissing now!!
SpiderMonkey has had noSuchMethod for years. The "Ten years"
complaint you make seems to be against "the ECMAScript committee", but
the committee mothballed the old ES4/JS2 proposals in 2003. Why?
Because of monopoly stagnation of the web by one company, not by the
committee. By pretending the committee is to blame you let that
company off the hook and perpetuate the problem you're decrying. That
doesn't make sense.
But if your only solution is to use a code generator, great. Don't
expect code generation to become a focus of the committee, however.
It's more likely to be addressed by competition among implementations,
as you note later:
Is it inevitable that browser makers and/or the ECMAScript standard group will add features to ECMAScript specifically to make it a better compilation target or expose actual byte code? By making ECMAScript engines faster, browser makers have already started. Should the effort to make the browser a better compilation target be encouraged and expedited so the long drawn out "eventually" can come sooner?
Why do you assume something not in evidence? Who says "eventually" or
"inevitabl[y]" there will be standard bytecode? Even if you end up
accurately predicting some future event, doing so prematurely based on
a hunch does not help us now in any way. Projecting enthusiastically
about hypothetical outcomes does not accelerate the process of
developing a suitable common language for hand-coders and code-
generators.
The JVM bytecode is a counter-example and we are not going to
standardize anything like it in the near term (next few years).
On Feb 12, 2009, at 9:17 PM, Brendan Eich wrote:
The JVM bytecode is a counter-example and we are not going to
standardize anything like it in the near term (next few years).
I meant by "counter-example" an example of what not to do. Same goes
for SWF ABC (used by Flash), which Adobe does not want to freeze and
standardize. These bytecode forms require verification, bind to VM
designs (or just bad old stack machine and typed operand thinking),
bring in unwanted patent concerns, and terminate in nothing
standardizable.
What's more, how many years (over 10?) has it been waiting for
invokedynamic, from a single vendor who didn't need to gain multi-
vendor standards-body consensus? Again if method_missing floats your
boat, you have solutions (native for some browsers, translated for the
rest), but bytecode is not any part of a solution for better language
features. It's more an impediment.
On Thu, Feb 12, 2009 at 9:17 PM, Brendan Eich <brendan at mozilla.com> wrote:
On Feb 12, 2009, at 8:07 PM, Peter Michaux wrote:
SpiderMonkey has had noSuchMethod for years. The "Ten years" complaint you make seems to be against "the ECMAScript committee",
Not at all and it is unfortunate it came across that way. It is at the whole chain of events which is long: "for the ECMAScript committee to standardize methodMissing, for browsers to implement it and for old browsers to disappear".
Is it inevitable that browser makers and/or the ECMAScript standard group will add features to ECMAScript specifically to make it a better compilation target or expose actual byte code? By making ECMAScript engines faster, browser makers have already started. Should the effort to make the browser a better compilation target be encouraged and expedited so the long drawn out "eventually" can come sooner?
Why do you assume something not in evidence? Who says "eventually" or "inevitabl[y]" there will be standard bytecode? Even if you end up accurately predicting some future event, doing so prematurely based on a hunch does not help us now in any way. Projecting enthusiastically about hypothetical outcomes does not accelerate the process of developing a suitable common language for hand-coders and code-generators.
I didn't make predictions. I asked questions on purpose.
If a "common language for hand-coders and code-generators" is desirable, isn't it necessary to consider the code-generator part? That was really the question of my whole post: should we be considering the code-generation part perhaps more than we are.
Peter
On Feb 12, 2009, at 9:41 PM, Peter Michaux wrote:
Not at all and it is unfortunate it came across that way. It is at the whole chain of events which is long: "for the ECMAScript committee to standardize methodMissing, for browsers to implement it and for old browsers to disappear".
Standardizing a target bytecode or even AST encoding would take no
less time, almost certainly more.
We should talk about methodMissing for Harmony.
The only consolation today, with renewed browser competition and self-
updating browsers and OSes, is that it ought not take as long to see
old user agent share drop as it did back in the bad old days.
Is it inevitable that browser makers and/or the ECMAScript standard group will add features to ECMAScript specifically to make it a
better compilation target or expose actual byte code? By making ECMAScript engines faster, browser makers have already started. Should the
effort to make the browser a better compilation target be encouraged and expedited so the long drawn out "eventually" can come sooner?Why do you assume something not in evidence? Who says "eventually" or "inevitabl[y]" there will be standard bytecode? Even if you end up accurately predicting some future event, doing so prematurely based
on a hunch does not help us now in any way. Projecting enthusiastically
about hypothetical outcomes does not accelerate the process of developing a suitable common language for hand-coders and code-generators.I didn't make predictions. I asked questions on purpose.
Sure, but your questions were loaded :-P.
If a "common language for hand-coders and code-generators" is desirable, isn't it necessary to consider the code-generator part?
I didn't agree that such a desert-topping/floor-wax was desirable. I
did suggest that implementations will optimize for workload offered by
the web, including the code generator part of it, better and sooner
(incrementally) compared to the standards process. And without bytecode!
That was really the question of my whole post: should we be considering the code-generation part perhaps more than we are.
I still say no, for reasons given. I don't think the tail (committee)
is wagging the dog (web content authoring by hand) here. If code
generators take over along with Skynet, we can change focus. But the
parts closer to the dog's brain (browser engines; dog meat :-P) will
precede the committee and inform it.
On Thu, Feb 12, 2009 at 10:16 PM, Brendan Eich <brendan at mozilla.com> wrote:
On Feb 12, 2009, at 9:41 PM, Peter Michaux wrote:
Not at all and it is unfortunate it came across that way. It is at the whole chain of events which is long: "for the ECMAScript committee to standardize methodMissing, for browsers to implement it and for old browsers to disappear".
Standardizing a target bytecode or even AST encoding would take no less time, almost certainly more.
We should talk about methodMissing for Harmony.
Great but to be clear, methodMissing was just an example of a feature some desire. I could have used "classes", "continuations", "threads", etc all of which people have tried to add to ES.
Peter
On Feb 12, 2009, at 10:23 PM, Peter Michaux wrote:
On Thu, Feb 12, 2009 at 10:16 PM, Brendan Eich <brendan at mozilla.com>
wrote:On Feb 12, 2009, at 9:41 PM, Peter Michaux wrote:
Not at all and it is unfortunate it came across that way. It is at
the whole chain of events which is long: "for the ECMAScript committee
to standardize methodMissing, for browsers to implement it and for old browsers to disappear".Standardizing a target bytecode or even AST encoding would take no
less time, almost certainly more.We should talk about methodMissing for Harmony.
Great but to be clear, methodMissing was just an example of a feature some desire. I could have used "classes", "continuations", "threads", etc all of which people have tried to add to ES.
Let's not go in circles or generalize until there's nothing to talk
about. I've argued against full continuatons, also against threads
(see my blog). Any particular responses?
Generators inspired by Python are in JS1.7 and 1.8 (generator
expressions), and proposed for harmony. One issue already raised:
should Harmony depart from Python and JS1.7 by requiring some new
keyword instead of 'function', say:
generator odds() { for (let i = 1; ; i += 2) yield i; }
I'm still inclined to follow Python and re-use 'function' as it reused
'def', for reasons given in the PEPs. People do not get confused by
the change yield in a function makes to the meaning of the function
(that it becomes a factory for generator-iterators). At least not in
our experience with JS1.7+, or AFAIK with Python.
Some kind of classes as sugar, probably zero inheritance, private
variables, public methods, probably public static methods are on the
boards for Harmony.
A methodMissing hook came up as a late request for 3.1. Allen
responded that it didn't fit in the schedule and suggested any such
feature should handle property accesses as well as method calls:
Igor pointed out how call and apply won't work with a noSuchMethod
mechanism tied to invocation:
You were perfectly clear in lamenting the time to get things
standardized and then implemented, and then to wait for old browsers
to die off (I share your lament), but there's no point in complaining
about this general problem, or laundry-listing good and bad (IMHO :-P)
features that may or may not have been implemented or proposed, to
justify code generators. If we're to improve the standard language for
either hand-coders or compilers, let's get down to brass tacks.
Brendan Eich wrote:
On Feb 12, 2009, at 9:41 PM, Peter Michaux wrote:
If a "common language for hand-coders and code-generators" is desirable, isn't it necessary to consider the code-generator part?
Yes.
I didn't agree that such a desert-topping/floor-wax was desirable.
Almost all programming languages are used as code generation targets, and that is a perfectly reasonable way of using them that should be considered as part of language design, including the design of ES-Harmony.
It is too late to consider significant changes to ES3.1 supporting its use as a code generation target.
I haven't digested the details of this thread. However, I did want to go
On Feb 13, 2009, at 7:45 AM, David-Sarah Hopwood wrote:
Brendan Eich wrote:
On Feb 12, 2009, at 9:41 PM, Peter Michaux wrote:
If a "common language for hand-coders and code-generators" is desirable, isn't it necessary to consider the code-generator part?
Yes.
I didn't agree that such a desert-topping/floor-wax was desirable.
Almost all programming languages are used as code generation targets, and that is a perfectly reasonable way of using them that should be considered as part of language design, including the design of ES- Harmony.
C is used as a code generation target, but it's portable assembler so
there's no issue.
JS is used by many more programmers, amateur and pro, than C. It has
to have better human factors than C. That goes against being a good
code generator target language.
It is too late to consider significant changes to ES3.1 supporting its use as a code generation target.
No, really? :-/
Brendan Eich wrote: ...
JS is used by many more programmers, amateur and pro, than C. It has to have better human factors than C. That goes against being a good code generator target language.
I totally agree with the first two sentences. I reserve judgment regarding the third.
On Fri, Feb 13, 2009 at 10:51 AM, Brendan Eich <brendan at mozilla.com> wrote:
JS is used by many more programmers, amateur and pro, than C. It has to have better human factors than C. That goes against being a good code generator target language.
I think that may not be true in all cases. methodMissing is an example where both groups could benefit. Tail calls is another feature I think benefit both especially as interest in functional languages is rising strongly.
When weighing the pros and cons of a proposed feature, knowing that putting "helps compilers" in the pros column is considered acceptable is important. Does "helps compilers" carry any weight at all with the committee? If it does then there is no need for anyone to dance around it for fear of being called a distractor or disruptively off topic. They can just say it as part of an acceptable discussion.
Peter
On Feb 13, 2009, at 11:07 AM, Allen Wirfs-Brock wrote:
Brendan Eich wrote: ...
JS is used by many more programmers, amateur and pro, than C. It has to have better human factors than C. That goes against being a good code generator target language.
I totally agree with the first two sentences. I reserve judgment
regarding the third.
Can you get off the fence on adding goto? How about call/cc?
(Lars added goto to make an old game port from C to Opera JS.)
On Feb 13, 2009, at 11:10 AM, Peter Michaux wrote:
On Fri, Feb 13, 2009 at 10:51 AM, Brendan Eich <brendan at mozilla.com>
wrote:JS is used by many more programmers, amateur and pro, than C. It
has to have better human factors than C. That goes against being a good code
generator target language.I think that may not be true in all cases.
I wrote "that goes against". It was not an exhaustive claim, it was
about tendency and design focus. When push comes to shove, it means no
goto, no call/cc, no low-level machine types (exception: ByteVector as
discussed).
methodMissing is an example where both groups could benefit.
We added noSuchMethod for TIBET, not for code generators. But I'm
bored with this straw man already.
Tail calls is another feature I think benefit both especially as
interest in functional languages is rising strongly.
I happen to agree, but these already foundered during ES4 development.
When weighing the pros and cons of a proposed feature, knowing that putting "helps compilers" in the pros column is considered acceptable is important. Does "helps compilers" carry any weight at all with the committee?
I can only speak for myself, but "helps compilers" is hard to judge
without compilers in hand. And IMO it has to lose to usability and
safety-last human factors for things like call/cc.
If it does then there is no need for anyone to dance around it for fear of being called a distractor or disruptively off topic. They can just say it as part of an acceptable discussion.
We are not subject to a moderator rejecting unacceptable discussion.
I'm disagreeing with you, to the extent I can get clear statements to
agree or disagree with, 'sall.
On Feb 13, 2009, at 11:23 AM, Brendan Eich wrote:
Tail calls is another feature I think benefit both especially as
interest in functional languages is rising strongly.I happen to agree, but these already foundered during ES4 development.
More recently, Waldemar objected to lambdas due to their completion-
value being a hazard. Suggestions to fix this involve adding explicit
"result-here" syntax. But people on the committee agreed that lambdas
as proposed in the wiki were "good for code generators". So usability
concerns can and do trump utility to compilers.
My dessert-topping/floor-wax point was that we may create a monster
trying to satisfy both parties, e.g. with lambdas. We could certainly
take a long time arguing in committee. So discussion focused on
avoiding such conflicts would be best (if there is a way to avoid such
conflicts).
Can you get off the fence on adding goto? How about call/cc?
For now, I only have problems, not solutions. However, I think it is debatable whether call/cc is more approachable to beginners (who admittedly don't write compilers) than goto. Call/cc is probably less of an attractive nuisance as many simply don't understand it. It also is likely to have a larger impact upon implementations.
On Feb 13, 2009, at 11:42 AM, Allen Wirfs-Brock wrote:
Can you get off the fence on adding goto? How about call/cc?
For now, I only have problems, not solutions.
Me too :-).
However, I think it is debatable whether call/cc is more
approachable to beginners (who admittedly don't write compilers)
than goto. Call/cc is probably less of an attractive nuisance as
many simply don't understand it. It also is likely to have a larger
impact upon implementations.
I agree with you on the relative costs of goto and call/cc. But I
still say neither should go in, never mind implementation burden,
because neither is appropriate to the human target audience, which is
going to remain very large for the Harmony timeframe.
Don't mind me, I'm eager to avoid unconstrained Harmony design. If
there's one thing I've learned from ES4 and 3.1 (if not other
projects), it's to make early cuts and deferments.
On Thu, Feb 12, 2009 at 10:16 PM, Brendan Eich <brendan at mozilla.com> wrote:
We should talk about methodMissing for Harmony.
I did try
esdiscuss/2008-November/008143
but nothing seems to have come of it yet
Peter
On Feb 13, 2009, at 4:43 PM, Peter Michaux wrote:
On Thu, Feb 12, 2009 at 10:16 PM, Brendan Eich <brendan at mozilla.com>
wrote:We should talk about methodMissing for Harmony.
I did try
esdiscuss/2008-November/008143
but nothing seems to have come of it yet
I finally found time to write up a proposal, sketchy and incomplete,
but ready for some ever-lovin' es-discuss peer review ;-).
Comments welcome.
I also captured these on the wiki page:
In the function defined for the example's invoke item the first argument to apply probably should be obj rather than peer.
I would be inclined to specify an additional argument (probably the first) for each handler function that would be passed the "this" value. Closure capture like is done in the example works but my intuition is that there will be situations where it would be handy to use the same catch-all descriptor for several objects.
I assume that if one of the properties is missing from the descriptor it simply has its default "not there" behavior.
Is there also an Object.getCatchAllDescriptor function?
Who can set them? Can catch alls be defined on an object if its [ [Extensible]] is false, etc.
Overall, I like this general design direction for catch alls.
Thanks, I incorporated the obvious fixes and left others as issues or
feedback items, and copied your sig once to credit your additions.
Two open issues remain:
On May 4, 2009, at 6:20 PM, Allen Wirfs-Brock wrote:
In the function defined for the example's invoke item the first
argument to apply probably should be obj rather than peer.
Possibly, but in the example peer[id] may be a function that insists
on |this| being bound to peer, not obj. If the function is pure, the
arguments flow in and a result comes back, and the catchall delegates
without mutating peer or obj. If the function mutates |this|, then the
effects can be recovered by calling other methods, or by property
access, which again is delegated without a wrapper.
I would be inclined to specify an additional argument (probably the
first) for each handler function that would be passed the "this"
value. Closure capture like is done in the example works but my
intuition is that there will be situations where it would be handy
to use the same catch-all descriptor for several objects.
Agreed, but why not use |this| instead of an extra (leading) parameter?
-----Original Message----- From: Brendan Eich [mailto:brendan at mozilla.com]
On May 4, 2009, at 6:20 PM, Allen Wirfs-Brock wrote:
In the function defined for the example's invoke item the first argument to apply probably should be obj rather than peer.
Possibly, but in the example peer[id] may be a function that insists on |this| being bound to peer, not obj. If the function is pure, the arguments flow in and a result comes back, and the catchall delegates without mutating peer or obj. If the function mutates |this|, then the effects can be recovered by calling other methods, or by property access, which again is delegated without a wrapper.
You're right it is probably situational. As I look at the example again, I'm uncertain about your intent for the invoke and construct properties of the catch all descriptor. My original assumption was that they would be invoked if obj (the object itself, not one of its properties) was invoked or new'ed. However, in that case there wouldn't be a property id to pass as an argument to the handler function, yet the function signatures for invoke and construct in the example have an id argument.
The other possibility, at least for invoke, is that the handler is invoked whenever an attempt is made to call the value of an non-existent property of obj. In other words invoke and get are mutually exclusive. I'm guessing that wasn't you intent as it really doesn't match the ES meta-model. However, I think I was momentarily thinking that way when I made the observation about passing obj rather than peer. That raises for me some usability concerns. I'm well aware that a "method" is called by first getting the property value and then invoking the function value returned from the get but I still made the mental slip of thinking that invoke in the catch all descriptor was about calling a method of obj rather than calling obj itself. I expect that the typical non-expert JavaSript programmer will be even more prone to making that mistake.
I would be inclined to specify an additional argument (probably the first) for each handler function that would be passed the "this" value. Closure capture like is done in the example works but my intuition is that there will be situations where it would be handy to use the same catch-all descriptor for several objects.
Agreed, but why not use |this| instead of an extra (leading) parameter?
Good point, it depend upon whether the model is that the entire descriptor is associated with obj as its catch all handler or whether the individual handler functions are separately associated with obj. If the entire catch all descriptor object is associated with obj then I would expect that when one of its methods is invoked that the this object would be the descriptor object rather than obj. On the other hand, if the descriptor object is just a way to aggregate a bunch of optional arguments to defineCatchAll (similar to the defineProperty pattern) then it is reasonable to assume that the individual handler functions are invoked as if they were methods of obj and thus the this value would be obj.
On May 4, 2009, at 10:46 PM, Allen Wirfs-Brock wrote:
-----Original Message----- From: Brendan Eich [mailto:brendan at mozilla.com]
On May 4, 2009, at 6:20 PM, Allen Wirfs-Brock wrote:
In the function defined for the example's invoke item the first argument to apply probably should be obj rather than peer.
Possibly, but in the example peer[id] may be a function that insists on |this| being bound to peer, not obj. If the function is pure, the arguments flow in and a result comes back, and the catchall delegates without mutating peer or obj. If the function mutates |this|, then
the effects can be recovered by calling other methods, or by property access, which again is delegated without a wrapper.You're right it is probably situational. As I look at the example
again, I'm uncertain about your intent for the invoke and construct
properties of the catch all descriptor. My original assumption was
that they would be invoked if obj (the object itself, not one of its
properties) was invoked or new'ed. However, in that case there
wouldn't be a property id to pass as an argument to the handler
function, yet the function signatures for invoke and construct in
the example have an id argument.
Yes, they are for "invoking a method" and "constructing via new
applied to a method" respectively. From
"One user expectation, fed and watered by E4X, is that methods are not
properties, or to avoid this JS oxymoron, that methods are not in the
default property namespace:
js> x = <room><width>12</width><length>16</length></room> <room> <width>12</width> <length>16</length> </room>
js> x.length
16 js> x.length()
1 Another point in favor of a call * catchall is efficiency:
var o = {
peer: ...,
get *(id) {
return function () { return this.peer[id](this.peer, arguments) }
}
}
creates a closure per get, whether or not the get result is called.
Whereas
var o = { peer: ..., call *(id, args) { return this.peer[id].apply(this.peer, args) } } creates a generic call forwarder."
It isn't just E4X. People want to make objects with arbitrary methods
callable on a given object, e.g. when bridging to CORBA or Java or
whatever. There may be static types in the peer space, but still you
don't want to mandate a custom JS object mapping exactly the M methods
in type T via hardcoded proxying. You don't want to reify the methods
as function objects stored in obj, even. You want to direct the call
through to the peer function.
This is why invoke has an ident leading parameter in proposals:catchalls .
Invoking a callable object is best done through a separate protocol,
not catchalls but something like the ES4 intrinsic::invoke (contrast
to meta::invoke the catchall hook for invoking a property, originally
not expressed by that qualified name but rather via dedicated 'call
*(id, args) {...}' syntax in the property initialiser production under
the object initialiser syntax).
The other possibility, at least for invoke, is that the handler is
invoked whenever an attempt is made to call the value of an non- existent property of obj.
The general intention for catchalls is to call the action hook only
when the property is missing. The odd case is set, vs. add in the
strawman (or first-set vs. every-set in the prose).
There's a less common case of wanting every-get too, to filter results
of and/or control access to existing properties.
In other words invoke and get are mutually exclusive. I'm guessing
that wasn't you intent as it really doesn't match the ES meta-model.
No, you've got it -- get and invoke are exclusive precisely to avoid
cons'ing up methods -- function-valued properties -- just to dispatch
calls.
However, I think I was momentarily thinking that way when I made
the observation about passing obj rather than peer. That raises for
me some usability concerns. I'm well aware that a "method" is called
by first getting the property value and then invoking the function
value returned from the get but I still made the mental slip of
thinking that invoke in the catch all descriptor was about calling a
method of obj rather than calling obj itself. I expect that the
typical non-expert JavaSript programmer will be even more prone to
making that mistake.
This is the non-mistake droid you are looking for, and what I was
trying to sketch. I'll clarify the strawman page. Thanks for bringing
this to light.
I would be inclined to specify an additional argument (probably the first) for each handler function that would be passed the "this" value. Closure capture like is done in the example works but my intuition is that there will be situations where it would be handy to use the same catch-all descriptor for several objects.
Agreed, but why not use |this| instead of an extra (leading)
parameter?Good point, it depend upon whether the model is that the entire
descriptor is associated with obj as its catch all handler or
whether the individual handler functions are separately associated
with obj. If the entire catch all descriptor object is associated
with obj then I would expect that when one of its methods is invoked
that the this object would be the descriptor object rather than
obj. On the other hand, if the descriptor object is just a way to
aggregate a bunch of optional arguments to defineCatchAll (similar
to the defineProperty pattern) then it is reasonable to assume that
the individual handler functions are invoked as if they were methods
of obj and thus the this value would be obj.
That's the idea -- like defineProperty, the object initialiser syntax
is a convenient approximation of named parameter passing. No this-
binding implied (as indeed none is implied in other object
initialisers).
On May 5, 2009, at 12:09 AM, Brendan Eich wrote:
"One user expectation, fed and watered by E4X, is that methods are
not properties, or to avoid this JS oxymoron, that methods are not
in the default property namespace: js> x = <room><width>12</width><length>16</length></room> <room> <width>12</width> <length>16</length> </room> js> x.length 16 js> x.length() 1 [snip]
The other possibility, at least for invoke, is that the handler is
invoked whenever an attempt is made to call the value of an non- existent property of obj.The general intention for catchalls is to call the action hook only
when the property is missing.
This is certainly true of noSuchMethod from SpiderMonkey, inspired
by doesNotUnderstand in Smalltalk (which spawned method_missing in
Ruby), but the E4X <room> snipped above shows a counter-example.
There, even though 'length' names a "property" (E4X is so strange that
the scare quotes are justified), we want to treat invocation of length
as different from a get followed by a call of the got value.
The odd case is set, vs. add in the strawman (or first-set vs. every- set in the prose).
That's one obvious exception. I clarified in the wiki how has, get,
and invoke may also want to be called only for non-existent
properties. Or not, if you are implementing E4X.
Even delete might want to be called for non-existent properties, to
fake its boolean result based on existence of a peer property not yet
reified.
There's a less common case of wanting every-get too, to filter
results of and/or control access to existing properties.
This case we have had in SpiderMonkey's C API forever. It's generally
a hazard where careless embedders end up running every property get
through a class getter that does nothing useful and just slows down
the access. We have to work around it in TraceMonkey when JITting.
We'd like to break the API here, soon.
Not sure how much of a lesson this gives, but I suspect every-get is
going to be the uncommon case for catchall users.
Still, as noted for delete above, full virtualization might want
every- and first- forms of all of the hooks. Instead of the add/set
split I wrote up, we could drop add and split all the hooks in two,
burdening the uncommon case with the longer name (everySet and set;
blech). I'm resisting this, it feels overdesigned, besides making for
ugly names.
Comments from Caja, Web Sandbox, and other such folks are more than
welcome here.
On Mon, May 4, 2009 at 2:34 PM, Brendan Eich <brendan at mozilla.com> wrote:
I finally found time to write up a proposal, sketchy and incomplete, but ready for some ever-lovin' es-discuss peer review ;-).
Comments welcome.
Thanks for posting your proposal. I agree with Allen Wirfs-Brock: at first look, this seems like a nice general design direction giving the programmer control over all the different otherwise internal methods on an object.
Peter
On May 5, 2009, at 12:26 AM, Brendan Eich wrote:
Still, as noted for delete above, full virtualization might want
every- and first- forms of all of the hooks. Instead of the add/set
split I wrote up, we could drop add and split all the hooks in two,
burdening the uncommon case with the longer name (everySet and set;
blech). I'm resisting this, it feels overdesigned, besides making
for ugly names.
The current design tries to distinguish hooks run for every access
(whether or not the property exists) from those run only if the
property does not exist:
hit or miss: get, set, delete miss only: has (before get), add (before set) miss at least: invoke, construct
Thus has : get :: add : set. If you want a hook that is called when a
property not in the object is about to be the subject of a get, use
has. If you want to hook into property creation via set of an id for
which no property exists in the object, use add. In any case, get and
set are always called.
The symmetry breaks because DefaultAction for set is to create the
missing property, whereas for get it is to return undefined. But this
symmetry break could help minimize the API and avoid a "has" hit/miss
hint arguments, getHit/getMiss, etc., hooks.
Because of its boolean result, delete should probably be called
whether or not the property exists. Again this design intentionally
shies away from deleteHit/deleteMiss hook bloat.
The case for invoke based on noSuchMethod/doesNotUnderstand/
method_missing is analogous to has for get and add for set. But again
this does not allow self-hosting E4X (something you may not care
about, but I do, since SpiderMonkey's E4X implementation should be
self-hosted to reduce code size and attack surface) and things like
it. There's no corresponding "hit or miss", always-called hook for
invoke, if we stick to this course.
I surmise that construct is in the same category as invoke, but I
could be wrong.
Alternative scheme, which still avoids requiring method cons'ing/ memoizing:
hit or miss: get, set, delete, invoke, construct
miss only: has, getMiss, setMiss, deleteMiss, invokeMiss,
constructMiss
Rationale: has is still useful when we do want to reify for whatever
reason; invokeMiss is noSuchMethod; the short names work for the
universal if not always-most-common cases.
My comfort level in this starts to fall significantly as we get into the hit or miss space. That is the point it suddenly feels like it is trying to be (or should be trying to be) a complete meta-object protocol rather than just a reification of failed property accesses. The former is interesting but I think it is much harder to justify, design, and efficiently implement than the latter. For now, I'm not convinced that we have a compelling need for the full meta-object protocol. If nothing else designing a simple catch all handler model is good practice for designing a full MOP.
Other than creeping toward being a MOP, my main concern is that the current design proposal just feel like it is getting too complex (needing to know which are hit or miss only operations, needing to understand how language level operations decompose into sequences of meta level operations). In addition, I think the terminology should better reflect the common user level understanding of the language.
A simpler model would only have these handlers assign called on any explicit assignment to a non-existent property invoke called on any call to a non-existent property construct called anytime new is used with a non-existent property delete delete a non-existent property getValue called on any implicit or explicit non-existent property access to that is not one of the above
At most one of the above is called on any property reference, and only if the property does not exist. I thought a bit about various schemes for dealing with catch alls along the prototype chain but the complexity rapidly grows. I concluded that it is probably better to just let a handler set implement its own prototype lookup logic if it wants to allow for prototype inheritance of catch all provided properties (that also supplies a use case for Object.create(null))
Does this provide enough to support your E4X scenario?
On May 5, 2009, at 3:09 PM, Allen Wirfs-Brock wrote:
My comfort level in this starts to fall significantly as we get into
the hit or miss space. That is the point it suddenly feels like it
is trying to be (or should be trying to be) a complete meta-object
protocol rather than just a reification of failed property accesses.
The former is interesting but I think it is much harder to justify,
design, and efficiently implement than the latter. For now, I'm not
convinced that we have a compelling need for the full meta-object
protocol. If nothing else designing a simple catch all handler model
is good practice for designing a full MOP.
Agreed, but here we are. Meet Scott Isaacs and Web Sandbox :-).
We need to be pragmatic or the design could be cut down too far. I
completely agree that we shouldn't slide down the slippery slope; my
"Art of the Meta-Object Protocol" is in a box somewhere for a
reason ;-). But real-world use-cases are pressing upon us.
Other than creeping toward being a MOP, my main concern is that the
current design proposal just feel like it is getting too complex
(needing to know which are hit or miss only operations, needing to
understand how language level operations decompose into sequences of
meta level operations). In addition, I think the terminology should
better reflect the common user level understanding of the language.
That would prefer "get" or "getValue" (or [[Get]] or GetValue? :-P).
The spec language and common user-level understanding are miles if not
light-years apart.
A simpler model would only have these handlers assign called on any explicit assignment to a non-existent property
Still does not connote "non-existent".
invoke called on any call to a non-existent property
I am with you, although the above objection stands, and
doesNotUnderstand, method_missing, and noSuchMethod (ignoring __
ugliness) all have their strong points.
construct called anytime new is used with a non-existent property
Ditto last comment.
delete delete a non-existent property
Sure (hard to care about this case), but the ability to use catchalls
to emulate arrays may be an important use-case for virtualizing
systems. If delete can't be handled for existent properties then
length can't be updated.
getValue called on any implicit or explicit non-existent property
access to that is not one of the above
What is an implicit property access other than the above? I added
"has" because property lookup does not getValue, and must not, but
unless you ignore catchalls on the global object and objects named by
with statement heads, you need "has" too.
At most one of the above is called on any property reference, and
only if the property does not exist.
Besides making it impossible to virtualize arraylikes, this will make
virtual DOMs hard to implement efficiently.
I thought a bit about various schemes for dealing with catch alls
along the prototype chain but the complexity rapidly grows.
Yes, we do not want to go up the proto chain twice, or any such thing.
I concluded that it is probably better to just let a handler set
implement its own prototype lookup logic if it wants to allow for
prototype inheritance of catch all provided properties (that also
supplies a use case for Object.create(null))
Right, that was the ES4 design's decision too.
But we need DefaultAction to avoid recursive divergence on the direct
("own") object.
Does this provide enough to support your E4X scenario?
No, but thinking about it more, that scenario needs invoke to be
called not only for missing property reference, but also for non-
callable value in an existing property. See the <room> example. That
is not how noSuchMethod works in SpiderMonkey:
js> o = {p:1, noSuchMethod: function (id, args) { print(id,
uneval(args)); }} [object Object] js> o.p
1 js> o.p()
typein:3: TypeError: o.p is not a function js> delete o.p
true js> o.p()
p [] js> o.p(1,2,3)
p [1, 2, 3]
So perhaps invoke could be for non-callable (including non-existent,
i.e., imputed-undefined-value) properties.
-----Original Message----- From: Brendan Eich [mailto:brendan at mozilla.com] ... my "Art of the Meta-Object Protocol" is in a box somewhere for a reason ;-). But real-world use-cases are pressing upon us.
I dug mine out after the last F2F, it's been sitting on my desk unread since...
A simpler model would only have these handlers assign called on any explicit assignment to a non-existent property
Still does not connote "non-existent".
If all these handlers are only invoked in missing property situations then we can probably get away with an implicitly non-existent connotation.
...
delete delete a non-existent property
Sure (hard to care about this case), but the ability to use catchalls to emulate arrays may be an important use-case for virtualizing systems. If delete can't be handled for existent properties then length can't be updated.
So, maybe delete falls into a different category of handlers (perhaps along with invoke and construct on the object itself, defineProperty, etc...) that aren't about non-existent properties but are hooks on important object level meta operations (at MOPish step but it keeps the catch all story cleaner).
getValue called on any implicit or explicit non-existent property access to that is not one of the above
What is an implicit property access other than the above? I added "has" because property lookup does not getValue, and must not, but unless you ignore catchalls on the global object and objects named by with statement heads, you need "has" too.
I hadn't thought through "implicit" but I was at least thinking about accesses from built-ins. I would also expect that getValue (and the others) would need to work on implicit accesses to the global object or a with object. With lookup might be trickly but I think it could be probably be make to work at least in the specification. It might be necessary to make DefaultAction contextual so that it does something different when trying to resolve a with binding
At most one of the above is called on any property reference, and only if the property does not exist.
Besides making it impossible to virtualize arraylikes, this will make virtual DOMs hard to implement efficiently.
Is there anything other than the delete hander that is needed to emulate arrays? What is the DOM problem?
...
So perhaps invoke could be for non-callable (including non-existent, i.e., imputed-undefined-value) properties.
I have some similar concerns about assignments to readonly properties and perhaps other error conditions. Maybe they are error handlers.
On a complete different plane, here is a wacky idea that integrates this work into Object.defineProperty: Object.defineProperty(obj, undefined, {invoke: function(id, args) {...}, ...});
Keeps the API surface smaller and emphasizes that this is really an way to define the handling of otherwise undefined properties.
On May 5, 2009, at 6:53 PM, Allen Wirfs-Brock wrote:
Still does not connote "non-existent".
If all these handlers are only invoked in missing property
situations then we can probably get away with an implicitly non- existent connotation.
The ones that want to be called for existent properties in order to
maintain an arraylike 'length' seem to be set and delete. For delete,
only for identifiers that are indexes (ToUint32(id) === id), which
could be a helpful fact, or a temptation to make distinctions on
identifiers that we should resist.
delete delete a non-existent property
Sure (hard to care about this case), but the ability to use catchalls to emulate arrays may be an important use-case for virtualizing systems. If delete can't be handled for existent properties then length can't be updated.
So, maybe delete falls into a different category of handlers
(perhaps along with invoke and construct on the object itself,
defineProperty, etc...) that aren't about non-existent properties
but are hooks on important object level meta operations (at MOPish
step but it keeps the catch all story cleaner).
'delete obj.prop' or 'delete obj[id]' needs to be caught and handled
by some catchall associated with obj, not the existing property value
of obj.prop or obj[id]. Again the arraylike case leads the way.
Invoking an object via obj(), constructing via new obj -- these cases
indeed lack a .prop or [id] to catch. They would want a lower-level MOP.
But obj.prop() is worth a catchall, this is the popular
noSuchMethod case, so that one does not have to create a method
for each value of 'prop' and retain it in obj under that name. The
call can be forwarded without any proxy or cloned method.
getValue called on any implicit or explicit non-existent property access to that is not one of the above
What is an implicit property access other than the above? I added "has" because property lookup does not getValue, and must not, but unless you ignore catchalls on the global object and objects named by with statement heads, you need "has" too.
I hadn't thought through "implicit" but I was at least thinking
about accesses from built-ins. I would also expect that getValue
(and the others) would need to work on implicit accesses to the
global object or a with object. With lookup might be trickly but I
think it could be probably be make to work at least in the
specification. It might be necessary to make DefaultAction
contextual so that it does something different when trying to
resolve a with binding
You can't getValue speculatively, the hook might have effects.
The ES4 thinking was not far off, if you squint past namespaces, from
what you're proposing, provided you add 'has' or 'hasProperty' to the
suite:
bugs.ecmascript.org/ticket/214
These catchalls (invoke as noSuchMethod is absent from this
ticket, but it fits the same mold as the others) run only when there's
no "own" property being acted upon.
Lars's comment 5 (bugs.ecmascript.org/ticket/214#comment:5) is
worth a read. The logic leading up to "Ergo no property managed by the
catchalls is ReadOnly" is still compelling given the practical
obstacles to defining ReadOnly properties on prototypes. Sure, with
ES5 you can make such non-configurable (else why bother?) constants on
a prototype, or make an object containing such constants be the
prototype of another object, but then [[CanPut]] comes into play. Best
to leave prototype-chasing to the catchall implementors, we agreed --
and so did this old ES4 ticket.
So apart from arraylikes (see below), I agree we should call catchalls
only for missing "own" property accesses.
At most one of the above is called on any property reference, and only if the property does not exist.
Besides making it impossible to virtualize arraylikes, this will make virtual DOMs hard to implement efficiently. Is there anything other than the delete hander that is needed to
emulate arrays? What is the DOM problem?
An arraylike would want set to monitor obj[31] = "foo" where
obj.length was < 32 (implying obj[31] did not exist before the
assignment), and similarly delete obj[31] should retract length when
obj[31] does exist.
And (here's a case where set would be wanted on an existing property)
obj.length = 30 should retract further. But why couldn't 'length' have
a custom setter, instead of trying to run a 'set' catchall on every
set just to intercept 'length' updates and extend or truncate?
DOM nodelists are "live" -- they are cursors into the shared DOM tree
and mutations can affect their contents. But they're also arraylike in
that they have .length and can be indexed. These could be implemented
(at some performance cost) by never creating "own" properties, instead
always searching the DOM, caching to optimize for no mutation as needed.
So, no need for catchalls on existing direct properties of the target
object, except for 'delete'. Is 'delete' the oddball catchall which
should be called for existing as well as missing direct properties?
So perhaps invoke could be for non-callable (including non-existent, i.e., imputed-undefined-value) properties.
I have some similar concerns about assignments to readonly
properties and perhaps other error conditions. Maybe they are error
handlers.
If only accessing a non-existent property were an error (in non-strict
mode)!
E4X would want something beyond the no-direct-property condition to
call the hook. The "out" for now would be the same drill as for DOM
nodelists: never populate direct properties, always compute results
via catchalls. I like it, it slows E4X down as a deprecation device, a
warning against using it :-P. (Not really kidding...)
On a complete different plane, here is a wacky idea that integrates
this work into Object.defineProperty: Object.defineProperty(obj, undefined, {invoke: function(id, args)
{...}, ...});Keeps the API surface smaller and emphasizes that this is really an
way to define the handling of otherwise undefined properties.
It's worth considering, but a little obfuscated (or just obscure --
undefined as id might be taken to convert to "undefined"). I don't
mind a separate defineCatchAll addition to Object, given the
successful additions in ES5. What do you say?
The other benefit of defineCatchAll is its atomic (re-)configuration
of catchalls. This is an implementation win where catchalls involve
switching a poor-man's vptr in the instance, or some similar identity-
preserving aggregation operation.
All -
On May 6, 2009, at 1:15 AM, Brendan Eich wrote:
But obj.prop() is worth a catchall, this is the popular
noSuchMethod case, so that one does not have to create a method
for each value of 'prop' and retain it in obj under that name. The
call can be forwarded without any proxy or cloned method.
Indeed. As the original you-know-what disturber that got the whole
noSuchMethod ball rolling, I would argue that having to do so
would dilute almost all of the value. It is precisely because
noSuchMethod is a catch all, for known and unknown properties
(especially important in a language where new properties can be
introduced on-the-fly), that makes it so valuable.
I don't think there is any disagreement that obj.prop() is the baseline use case for catchalls that must be supported. The original context of Brendan's comment was more about whether a handler was needed for obj() in addition to obj.prop().
Ah - right.
BTW, I'm just pleased as punch that this is being discussed for
Harmony - thanks guys!
Allen Wirfs-Brock wrote:
My comfort level in this starts to fall significantly as we get into the hit or miss space. That is the point it suddenly feels like it is trying to be (or should be trying to be) a complete meta-object protocol rather than just a reification of failed property accesses. The former is interesting but I think it is much harder to justify, design, and efficiently implement than the latter.
I disagree strongly. Splitting the handlers into "always" and "missing" versions is only a little more complex to specify, implement, and understand. The semantics that would be obtained by doing this are perfectly clear AFAICS.
Without this functionality, the proposal is much less useful for the secure JS subsets.
About five months ago I posted here about improving ECMAScript as a compilation target.
esdiscuss/2008-September/007652
I was disappointed that there weren't any responses to that email. Folks were probably busy with other concerns. I also thought there might be some motivation to avoid making ECMAScript an improved compilation target as that may reduce the number of programmers working directly in ECMAScript. Perhaps that goes against the goals of this group.
The general browser scripting world is still programming in ECMAScript; however, the number of projects that compile to ECMAScript are increasing. I expect to see this trend continue to grow.
I wrote the following as a blog post recently. I don't presume anyone here reads my blog so am submitting it here. I am interested to know if attitudes have changed or people are warming up to the idea of changing ECMAScript intentionally to make it a better compilation target.
The JVM is now a compilation target for many dynamic languages. The implementers of those dynamic languages have been waiting for invokedynamic so they can make their dynamic languages compile to more efficient JVM byte code. Sun seems to have caught on that people want to use the JVM but not use Java and Sun seems to be looking at ways to accommodate the compilation of these dynamic languages. Microsoft's .NET CLR+DLR is ahead of Sun and has implemented features for efficient dynamic language compilation. Developers have made it clear that even if they are locked into the platform, they are going to still use their language of choice.
The same is happening in the browser. Developers are compiling many languages down to ECMAScript ("they byte code of the browser"). GWT and Cappuccino are two well know examples. These projects are overruling the idea that they must write in ECMAScript to create browser apps. They don't have to wait for the ECMAScript committee to standardize methodMissing, for browsers to implement it and for old browsers to disappear. Ten years? No way! They have methodMissing now!!
Is it inevitable that browser makers and/or the ECMAScript standard group will add features to ECMAScript specifically to make it a better compilation target or expose actual byte code? By making ECMAScript engines faster, browser makers have already started. Should the effort to make the browser a better compilation target be encouraged and expedited so the long drawn out "eventually" can come sooner?
Peter
About five months ago I posted here about improving ECMAScript as a compilation target. https://mail.mozilla.org/pipermail/es-discuss/2008-September/007652.html I was disappointed that there weren't any responses to that email. Folks were probably busy with other concerns. I also thought there might be some motivation to avoid making ECMAScript an improved compilation target as that may reduce the number of programmers working directly in ECMAScript. Perhaps that goes against the goals of this group. The general browser scripting world is still programming in ECMAScript; however, the number of projects that compile to ECMAScript are increasing. I expect to see this trend continue to grow. I wrote the following as a blog post recently. I don't presume anyone here reads my blog so am submitting it here. I am interested to know if attitudes have changed or people are warming up to the idea of changing ECMAScript intentionally to make it a better compilation target. ---- The JVM is now a compilation target for many dynamic languages. The implementers of those dynamic languages have been waiting for invokedynamic so they can make their dynamic languages compile to more efficient JVM byte code. Sun seems to have caught on that people want to use the JVM but not use Java and Sun seems to be looking at ways to accommodate the compilation of these dynamic languages. Microsoft's .NET CLR+DLR is ahead of Sun and has implemented features for efficient dynamic language compilation. Developers have made it clear that even if they are locked into the platform, they are going to still use their language of choice. The same is happening in the browser. Developers are compiling many languages down to ECMAScript ("they byte code of the browser"). GWT and Cappuccino are two well know examples. These projects are overruling the idea that they must write in ECMAScript to create browser apps. They don't have to wait for the ECMAScript committee to standardize methodMissing, for browsers to implement it and for old browsers to disappear. Ten years? No way! They have methodMissing now!! Is it inevitable that browser makers and/or the ECMAScript standard group will add features to ECMAScript specifically to make it a better compilation target or expose actual byte code? By making ECMAScript engines faster, browser makers have already started. Should the effort to make the browser a better compilation target be encouraged and expedited so the long drawn out "eventually" can come sooner? ---- Peter